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PREFACE 



Involving key stakeholders — particularly project 
directors and evaluators — as partners in the evaluation of 
character education programs is critical to demonstrating 
their usefulness and improving their effectiveness. In fact, 
recognizing the importance of mobilizing — marshalling 
people and other resources for action in support of a 
proposal — was a principal outcome of the Listening Ses- 
sion for Evaluation convened on March 1 1—12, 2004, 
by the U.S. Department of Education and the Character 
Education and Civic Engagement Technical Assistance 
Center (CETAC).' Participants at the session agreed that 
mobilizing a collaborative team to assist in evaluation 
would enhance each phase of the assessment process and 
provide greater understanding among all stakeholders, 
especially with respect to 

★ the evaluation standards set forth in the No 
Child Left Behind Act of 2001 and the Partner- 
ships in Character Education Program (PCEP) 
grant guidelines; 

★ unfamiliar evaluation terms (e.g., data-based 
decision-making. Institutional Review Board, 
contamination) that presented barriers in com- 
municating with evaluators; and 

★ key issues in conducting scientifically based 
evaluations of PCEP grants. 

PURPOSE AND DEVELOPMENT 
OF THE EVALUATION GUIDE 

Conducting scientifically rigorous evaluations of 
character education interventions is complex. The nature 
of character education compounds the typical challenges 
of evaluation in particular ways. This evaluation guide is 
presented as a resource primarily for project directors who 
are federal grantees embarking on an evaluation of a char- 
acter education intervention, although it contains useful 
information that can benefit other education administra- 
tors who also are providing these interventions. It offers 
strategies for working with external evaluators and key 
stakeholders in planning and implementing a scientifically 
sound evaluation. 



The guide is organized in a logical sequence that 
reflects the order in which to undertake the eight basic 
steps of planning and implementing an evaluation. The 
introduction explores the federal mandate for evaluation 
and notes the many ways that evaluation can contribute 
to the improvement, recognition and sustainability of an 
intervention. In addition to the list of references at the 
end of this report, there is a list of published resources at 
the end of each step. The guide also provides appendices 
with pertinent federal regulations, sample consent letters, 
a checklist of evaluation activities, examples for displaying 
data, and a glossary of common evaluation terminology. 
Finally, all of the Web sites throughout the report were 
last accessed Aug. 8, 2007. 

Knowledge alone is not sufficient to manage an effec- 
tive evaluation. As Jaeger (1990) has observed, evaluation 
in an education setting compels stakeholders to focus on 
the desire for school improvement, to become a part of 
collegial working relationships, and to be vigilant with 
details. These, of course, are qualities that many educators 
naturally bring to the task. 



1 . In Fiscal Year 2004, the CETAC was operated through a contract 
awarded to Caliber Associates (Contract No. ED-03-PO-2981). Two 
subcontractors supported Caliber: the Association for Supervision and 
Curriculum Development and the Character Education Partnership. In 
September 2004, the Pacific Insfitufe for Research and Evaluation was 
awarded the CETAC contract (No. ED-04-CO-0072/0001). 
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INTRODUCTION 



Many educators believe that implementing character 
education in their schools helps students develop ethically, 
socially and academically. Character education is an in- 
clusive term embracing all aspects of how schools, related 
social institutions and parents can support the positive 
character development of children and adults. The term 
character includes the emotional, intellectual and moral 
qualities of a person or group as well as the demonstration 
of these qualities in prosocial behavior. Relevant virtues 
include honesty, justice and fairness, trustworthiness, 
responsibility, respect, altruism, patience, perseverance, 
appreciation of diversity, and courage. The related devel- 
opment of moral reasoning, problem solving and inter- 
personal skills, a work ethic, empathy, and self-reflection 
is recognized as essential for optimal character develop- 
ment. For a school to foster character development, it 
must provide a positive social environment, characterized 
by leadership; collegiality; a learning orientation among 
faculty; and ties among school, home and community. 
Finally, practicing the virtues of civic engagement, civility, 
and citizenship and embracing the values of democracy 
are necessary for developing character in both the child 
and the community. 

A BRIEF HISTORY OF THE PARTNERSHIPS 
IN CHARACTER EDUCATION PROGRAM 

The U.S. Congress, recognizing the importance of 
character education, authorized the Partnerships in Char- 
acter Education Pilot Projects in 1994. Under this grant 
program, the secretary of education could make up to 
10 grants annually to state education agencies (SEAs) in 
partnership with one or more local educational agencies 
(LEAs). Between 1995 and 2001, 46 grants, representing 
more than $45 million, were awarded to SEAs to help 
communities organize a character education response to 
their own most compelling issues. This money (a) sup- 
ported the development of character education materials 
and their integration into the broader curriculum; (b) 
provided professional training for teachers; (c) facilitated 
the involvement of the parents, students and community 
in the design and implementation of their grant; and (d) 
required a comprehensive evaluation of the program. In 
fiscal year 2002, Congress reauthorized the PCEP as part 
of the Elementary and Secondary Education Act of 1965 
(ESEA), as amended by the No Child Left Behind Act of 
2001, and funding was expanded from $8 million to 
$25 million. 



EVALUATION REQUIREMENTS OF THE 
NO CHILD LEFT BEHIND ACT 

Under No Child Left Behind (NCLB), both SEAs and 
LEAs are eligible to apply for funding, and the evaluation 
requirement has taken on a new emphasis. Grant projects 
are required “to provide information that demonstrates 
that the program for which the grant is sought has clear 
objectives that are based on scientifically based research” 
{NCLB Section 5431 [e] [2] [A]). Once funded, programs 
are required to undergo periodic evaluations to assess their 
progress. The statute encourages research into the faithful- 
ness of implementation of the project and “evaluation of 
the success of fostering the elements of character selected 
by the recipient” {NCLB Section 5431 [b] [1] [C]). Funds 
may also be used to measure the integration of character 
education into both the curriculum and teaching methods 
of the school {NCLB, Section 5431 [b] [1] [b]), both of 
which should be evaluated for effectiveness. This guide 
is meant to help SEAs and LEAs meet the evaluation 
requirements. 

THE CHALLENGE OE SCIENTIFIC EVALUATION 

The federal mandate to undertake scientifically rigor- 
ous evaluation poses special challenges for the directors 
and evaluators of character education interventions. First, 
little precedent has been set in the evaluation world for 
assessing the types of outcomes that character educa- 
tion promotes: establishing a caring environment among 
students and teachers as well as instilling a positive moral 
identity in students. Second, the unfamiliar vocabulary 
of evaluation has presented a real language barrier in 
communicating with evaluators and in reviewing resource 
materials, especially with respect to research methodol- 
ogy, statistical procedures, contamination of data, and 
data-driven decisions. Last, the Institutional Review 
Board (IRB) process and requirements (described in Step 
4) are not familiar to most project directors. Neverthe- 
less, they agree that high-quality scientific evaluation of 
character education can be accomplished and that both 
the processes and the outcomes of evaluation would yield 
valuable information for strengthening character educa- 
tion interventions. 
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THE DEPARTMENT OF EDUCATION’S 
INSTITUTE OF EDUCATION SCIENCES 

The Education Sciences Reform Act of 2002 established 
within the U.S. Department of Education the Institute 
of Education Sciences (lES). The mission of lES is to 
provide rigorous evidence on which to ground education 
practice and policy (see http://ies.ed.gov). 

In 2002, lES established the What Works Clear- 
inghouse (WWC) to provide educators, policymakers, 
researchers and other interested parties with a central and 
trusted source of what works in education (see http: 

// WWW. wharworks . ed . gov) . 

According to lES, “[S]cientifically based research: 

★ employs systematic, empirical methods that 
draw on observation or experiment; involves 
data analyses that are adequate to support the 
general findings; relies on measurements or 
observational methods that provide reliable 
data; makes claims of causal relationships only 
in random-assignment experiments or other 
designs (to the extent such designs substantially 
eliminate plausible competing explanations for 
the obtained results); 

★ ensures that studies and methods are presented 
in sufficient detail and clarity to allow for rep- 
lication or, at a minimum, to offer the oppor- 
tunity to build systematically on the findings of 
the research; 

★ obtains acceptance by a peer-reviewed journal 
or approval by a panel of independent experts 
through a comparably rigorous, objective and 
scientific review; and 



EVALUATION OF CHARACTER 
EDUCATION PROGRAMS 

Program evaluations that are grounded in scientifi- 
cally based research add to our shared knowledge base 
and assist in making major advances in improving the 
effectiveness of American education. In particular, those 
evaluations may help to 

★ provide data to determine whether an interven- 
tion is accomplishing its desired objectives; 

★ support decision-making, guide practice and 
improve programming; 

★ nurture staff, student, parent and community 
efforts; 

★ communicate to parents and the community 
the purpose of the program and the benefits for 
the participants during the various stages of its 
implementation; 

★ inform funders about the outcomes of their 
investments; 

★ influence program and policy decisions; and 

★ build the knowledge base about what does and 
does not work in character education. 

Now that the why of evaluating character education 
interventions has been clarified, the next eight chapters 
detail the eight steps of program evaluation. The follow- 
ing resource listing provides sources of information for 
understanding character education evaluations that have 
been completed in recent years. 



★ uses research designs and methods appropriate 
to the research question posed” (USED/IES, 
WWC). 



GENERAL RESOURCES 



Publications 

Berkowitz, M. W. 1 998. A primer for evaluating a character education 
initiative. Washington, D.C.: Character Education Partnership. 

Blum, R. 2005. A case for school connectedness. Education Leader- 
ship (Association for Supervision and Curriculum Development) 62 
(7): 16-20. 

Blum, R., and H. Libbey, eds. 2004. School connectedness: Strength- 
ening health and education outcomes for teenagers. Special issue, 
Journal of School Health 74 (5). See http://www.jhsph.edu 
/wingspread/ Septemberissue.pdf 

Davidson, M.L. 2000. A special theme section: Action research and 
character education. of Research in Education 10 (1): 32—61. 

Laud, L., and M.W Berkowitz. 1999. Challenges in evaluating 
character education pvo^vdims. Journal of Research in Education 9(1): 
66-72. 



Leming, J. 1993. In search of effective character education. Educa- 
tional Leadership 51 (3): 63-71. 

. 1997. Whither goes character education? Objectives, peda- 
gogy, and research in education pto^V 2 ims. Journal of Education 179 
(2): 11-34. 

Mathison, S. 2005. Encyclopedia of evaluation. Thousand Oaks, 
Calif: Sage. 

National Research Council. 2002. Scientific research in education. 
Washington, D.C.: National Academy Press. 

Power, F.C., A. Higgins, and L. Kohlberg. 1989. Lawrence Kohlbergs 
approach to moral education. New York: Columbia University Press. 
(An example of a single case study.) 

Rossi, P.H., M.W. Lipsey, and H.E. Freeman. 2004. Evaluation: A 
systematic approach. 7th ed. Thousand Oaks, Calif: Sage. 

Schaps, E., M. Watson, and C. Lewis. 1996. A sense of community 
is key to effectiveness in fostering character education. Journal of Staff 
Development 17 (2): 42—47. 

Shavelson, R.J., and L. Towne. 2002. Scientific research in education. 
Washington, D.C.: National Academy Press. 

Internet Resource 

What Works Clearinghouse (WWC) — In particular, see the WWC 
Intervention Reports in which WWC reviews studies on specific 
character education interventions. See http://www.whatworks.ed.gov 
/Topic.asp?tid=12&ReturnPage=default.asp. 
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STEP 1 

PARTNER WITH AN 
EVALUATOR AND EORM 
AN EVALUATION TEAM 



The first step in the evaluation process — and perhaps 
the most critical — is forming the evaluation team. Al- 
though the team should represent all of the stakeholders, 
the two key players are the project director and the evalu- 
ator. Together, they should agree on and clarify responsi- 
bilities as well as establish a working relationship that will 
facilitate clear, effective communication. 

FINDING A SKILLED EVALUATOR 

The project director should identify and, if possible, 
hire an evaluator during the earliest stage of preparing the 
grant application. This approach enables the evaluator to 
develop a sound design that includes appropriate out- 
comes of and methods for assessing the planned program. 
A well-developed design or plan can then be incorporated 
into the evaluation section of the proposal. 

The project director should consider taking the fol- 
lowing steps to identify and hire an evaluator: 

Identify the resources and requirements of the 
SEA or LEA that is sponsoring the character educa- 
tion initiative. In most cases, the project director will 
have the responsibility for locating and developing a 
relationship with a qualified external evaluator. However, 
some project directors will have access to and be required 
to use internal evaluation resources such as an in-house 
evaluation department or evaluator. Other project direc- 
tors may have the option of hiring an external evaluator 
only through a competitive bid process. In that case, be- 
coming familiar with the organization’s policies and pro- 
cedures for contracting with an evaluator will make the 
hiring process much more efficient. Regardless of whether 
the evaluator is external or internal, he or she should be 
independent, separated from program implementation, 
and without any vested interests in the results. 

Determine desired qualifications. The evaluator 
should have relevant advanced graduate training in one of 
the social sciences and evaluation methods and, preferably, 
experience not only in conducting program evaluation 
research but also in writing the evaluation section of suc- 
cessful proposals. The evaluator should be knowledgeable 



about the laws and regulations that can affect the evalua- 
tion, including the Department of Education regulations 
for the Protection of Human Subjects (34 CFR 97), 
the Family Educational and Privacy Rights Act {FERPA), 
and the Protection of Pupil Rights Amendment (PPRA). 
Information about FERPA and PPRA can be found in 
appendix A. 

Identify potential candidates. To identify a quali- 
fied evaluator, search the published character education 
literature, ask for recommendations from other character 
education projects, or contact a college or university, 
nonprofit organization, or research firm. The Institute of 
Education Sciences’ What Works Clearinghouse has es- 
tablished a register of education evaluators at its Web site 
http://www.wharworks.ed.gov. In addition, the American 
Evaluation Association provides an extensive list of evalua- 
tors on its Web site http://www.eval.org. 

Contact candidates to assess their expertise, 
credibility and interpersonal style. Request a curricu- 
lum vita or resume from all candidates, references from 
project directors with whom the candidate has worked in 
conducting evaluations, and a sample evaluation report. 
Ideally, identify at least two evaluators who (a) have broad 
knowledge about evaluation techniques and design, expe- 
rience in evaluating education interventions, and familiar- 
ity with the population to be assessed and (b) demonstrate 
good interpersonal and communication skills. A helpful 
tool for comparison shopping among evaluators is the 
Character Education Evaluation Rating Scale (Posey, 
Davidson, and Korpi 2003). 

Screen and rate candidates. Interview top candi- 
dates. Explore the evaluator’s track record of providing 
evaluations on time and on budget, including dealing 
with IRBs and parent permission forms as well as achiev- 
ing targeted return rates of data from schools, students, 
teachers and parents. Be prepared not only to discuss the 
details of the proposed character education program, in- 
cluding its target population, history, philosophy, content 
and goals, but also to explore what is needed to develop a 
sound, feasible and ethical evaluation. Find out whether 
the evaluator is willing and available to assist you within 
the time frame needed. 

Select the final candidate. Choose the candidate 
who offers the best combination of evaluation expertise 
and potential for maintaining a positive working 
relationship. If you have candidates with comparable 
qualifications, then choose the one who is most accessible. 




MORTUZTNG 

for EVIDENCE-BASED 

CHARACTER EDUCATION 



Proximity will help you maintain face-to-face contact dur- 
ing the program’s implementation. 

ASSEMBLING A COLLABORATIVE 
ADVISORY EVALUATION TEAM 

A collaborative team, formed to advise and support 
evaluation, should include representatives from all stake- 
holder groups, including not only the school administra- 
tors, teachers, parents, students and community memhers 
hut also the evaluator, project director and program or 
intervention staffs Involving these stakeholder groups 
will help them buy into the evaluation activities and 
will help to focus the evaluation on the program’s goals 
and activities. Collaborating helps engage the stakehold- 
ers so they have the opportunity to express their goals 
for the project and understand how program outcomes 
and decision-making are connected to the evaluation. 

A collaborative process gives the stakeholders a more 
complete understanding of how outcomes are measured, 
which enables them to make better use of the findings. In 
fact, the evaluator is responsible for facilitating processes 
and teaching program staff members about evaluation. 
Engaging stakeholders in a collaborative process creates a 
schoolwide culture that is committed to ongoing learning 
through evaluation. 

ROLES AND RESPONSIBILITIES OF THE 
PROJECT DIRECTOR AND THE EVALUATOR 

The project director and the evaluator have distinct 
functions. The project director is responsible for ensur- 
ing that the evaluator understands the program and the 
context in which it operates by explaining its objectives, 
the mechanisms by which it achieves objectives, and the 
populations served. The project director must also sup- 
port the evaluator and the evaluation by providing ready 
access to needed data, records, personnel, stakeholders, 
and so forth. The project director should develop a writ- 
ten contract with the evaluator, which should include a 
description of evaluation tasks and products, a timeline 
and a budget. 

The evaluator works for the project but is not an 
advocate for it or for the program chosen. Evaluators have 
a professional responsibility to be objective about program 
strengths and weaknesses, to report their findings based 



on valid and reliable measures and, if necessary, to design 
questions and methods responsive to unique program 
goals. In addition, they must represent the interests of all 
stakeholders, not only program management. 

Evaluators bring experience in data analysis, survey 
development, research design and proposal writing. With 
this experience, they can help program staff members to 
(a) identify critical evaluation questions; (b) use evalua- 
tion data to make decisions about practices; and (c) com- 
municate the evaluation results to school administrators, 
the community and potential funders. 

The director and evaluator should both be thorough- 
ly familiar with PCEP evaluation and reporting require- 
ments. Each project director and evaluator will need to 
work together to develop a timeline and specify their re- 
spective responsibilities to fit the particular characteristics 
of the intervention and context. Exhibit 1 presents a typi- 
cal division of responsibilities between the project director 
and evaluator; however, individual grant projects may dif- 
fer and thus require a different breakdown of obligations. 
Responsibility for the evaluation should remain with the 
evaluator and, ultimately, with the funding agency. 



2. Implementing the program may be done by intervention staff and/or 
teaohers and school personnel. 



EXHIBIT 1 

RESPONSIBILITIES OF PROJECT DIRECTOR AND EVALUATOR 



ROLE 


PROJEOT DIREOTOR 


EVALUATOR 


LEADERSHIP 


Contract with the evaluator, following required 
policies and procedures for contracting, and 
establish a productive working relationship 
among stakeholders. Communicate program 
expectations to all stakeholders. 

Collaborate with evaluator to develop the 
written program description according to grant 
application standards. 

Inform the evaluator about the populations to 
be served and sensitive issues in implementing 
the evaluation. 

Plan for obtaining broad representation of 
parents and community. 

Lead and maintain the partnership among key 
stakeholders. 

Keep project staff members and control or 
comparison group participants informed about 
the evaluation and their responsibilities. 


Collaborate with project director to develop 
the written program description according to 
grant application standards. 

Develop evaluation design consistent with 
program description and grant application 
standards. 

Consult with project director to ensure that 
the evaluation plan is consistent with state and 
local agency standards. 

Prepare and submit application for the ap- 
proval of an IRB. 

Design and pilot test measures or identify 
reliable and valid instruments for assessment, 
including parent and community measures. 


MANAGEMENT 


Manage project design, staffing and budget. 

Supervise project staff members to ensure that 
the intervention is implemented as intended. 

Coordinate daily activities of the project. 

Confer with evaluator on sampling and consent 
procedures. 

Coordinate data collection procedures. 

Work with evaluator to supervise evaluation 
activities of the staff, including data collection 
and field observations. 


Recruit and oversee data collectors; oversee 
informed consent process. 

Train project staff members on research ethics 
and data collection procedures; prepare field 
observations. 

Ensure that all data collection procedures 
adhere to confidentiality requirements and data 
security. 

Maintain communication with the project 
director and attend team meetings as necessary. 

Implement data management and analysis 
procedures. 


REPORTING 


Present progress reports within state or local 
education agency. 

Prepare annual performance report. 

Present findings at local, national and 
international association meetings, as 
appropriate. 

Present findings in regional, national and 
international journals, as appropriate. 


Provide a feedback loop of information to 
project director in timely progress reports 
communicated in user-friendly language. 

Write annual evaluation reports for submission 
to the project director and funding agency. 

Present findings at local, national and 
international association meetings, as 
appropriate. 

Present findings in regional, national and 
international journals, as appropriate. 





MORTUZTNG 

for EVIDENCE-BASED 

CHARACTER EDUCATION 



RESOURCES FOR OBTAINING 
A QUAEIFIED EVAEUATOR 



American Evaluation Association — ^A source for locat- 
ing an evaluator and for obtaining evaluation publica- 
tions and information published by the association and 
its membership. See http://www.eval.org. 

Registry of Outcomes Evaluators, What Works 
Clearinghouse — ^An online database of professional 
evaluators who conduct research on the effects of 
educational interventions. See http;//www.whatworks 
.ed.gov/ technicalassistance/ overview.html. 




STEP 2 

DEVELOP A COMPREHENSIVE 
PROGRAM DESCRIPTION 



A program is a theory and an evaluation is its test. 
To organize the evaluation to provide a responsible 
test, the evaluator needs to understand the theoreti- 
cal premises on which the program is based. 

— Carol Weiss (1998, 55) 



Step 2 focuses on what should he included in the 
program description of a grant application. The program 
description presents the strengths of the chosen program, 
how it is expected to foster chosen outcomes, and how 
it fits with the schools and communities in which it will 
be implemented. The program description also lays the 
foundation for the evaluation plan. Writing the grant 
application proposal is the first important area of collabo- 
ration between the project director and the evaluator. The 
project director, with the collaboration of the evaluator, 
spells out the assumptions and goals of the program. For 
instance, the project director may make an assumption 
that values-based classroom discussions will affect the goal 
of fostering students’ values-based reasoning and problem- 
solving abilities. Then, the evaluator uses the program 
assumptions and goals to spell out the evaluation plan, 
including the research design and the measures that will 
be used to assess whether and how well the program goals 
have been met. The discussion here in Step 2 lays out how 
to think about and write a program description, and Step 
3 discusses writing the evaluation plan. 

CREATING A CLEAR AND COMPREHENSIVE 
PROGRAM DESCRIPTION 

When writing a grant application proposal to fund 
an intervention, the project director needs to clearly de- 
scribe what the program or intervention emphasizes, what 
it assumes, who its target audience is, and what its goals 
are. Generally speaking, character education interventions 
emphasize promoting character development, prosocial 
behavior and academic achievement in students. These 
interventions are usually based on the assumption that to 
accomplish those goals, the school should have a positive 
climate, the teachers should bring character issues into 
their teaching, and students should have opportunities 
to display both their character and academic strengths. 
The target audience of character education interventions 



is the students and, secondarily, the teachers, parents and 
community. The proposal should spell out these aspects of 
the intervention and how they are incorporated into the 
evaluation plan that will serve as a guide for data collec- 
tion and data analysis. In addition, the proposal should 
specify what will be assessed periodically during the grant 
period from its beginning until it is completed. 

A clear and convincing grant application proposal 
should describe the issues and problems the intervention 
seeks to address and why the chosen intervention is an 
effective way to address them. It should specify program 
goals and explain how the evaluation design will assess 
whether and how well the goals are met. The project 
director and evaluator should collaborate on this task of 
writing a proposal that includes a detailed description 
of the program or intervention and an evaluation plan, 
woven together logically into an effective narrative. 

ADDRESSING KEY AREAS IN THE 
PROGRAM DESCRIPTION 

This section offers an organizational structure for 
writing a program description. The five areas defined here 
are the usual necessary components of any grant applica- 
tion (although they may be labeled differently for various 
grants); context, goals, program requirements, broad char- 
acteristics, and intervention guidelines. The definitions 
and specific examples of program details that fall within 
each area focus on character education. Any one proposal 
may include some, but not necessarily all, of these areas 
and will likely also have additional program-specific areas 
to discuss. Each area lists possible elements to encourage 
the project director and evaluator to think through the 
details of the proposed program or intervention carefully. 

Context Area: Position the proposed program or 
intervention in relation to other character education 
programs and relevant research in character educa- 
tion. The context narrative should review findings about 
other existing programs that are widely used, demonstrat- 
ed to be effective by scientifically based research, or both. 

It should explain how the proposed program is similar to 
and different from these programs. In addition, it should 
describe the background of the proposed program and its 
history of use as well as related research. Because this area 
lays out the background of the program, all of the fol- 
lowing aspects should be addressed, at least to the extent 
possible; 

★ Background and history of the proposed program 

★ The relationship of the program to other pro- 
grams in character education 





MORTUZTNG 

for EVIDENCE-BASED 

CHARACTER EDUCATION 



★ Use of the program by other schools, districts 
and states 

★ Research findings about the effectiveness of 
the program 

★ Research findings about the effectiveness of 
similar programs 

★ Ways in which the proposed program is similar 
to and different from existing scientifically 
based programs 

★ Strengths of the proposed program 

Goals Area: Determine the program goals for all 
stakeholders. The overall purpose and the specific goals 
of the intervention should be stated in detail. The goals 
of the intervention specify what it is supposed to do, in 
other words, what outcomes are expected. The evaluator 
will use the goals to choose appropriate outcome mea- 
sures. It is important to define all program goals, not only 
for students but also for stakeholders other than students, 
including teachers, parents, administrators and the com- 
munity, for example: 

Goals for Students 

★ Develop prosocial attitudes 

★ Cultivate moral and values-based reasoning 
abilities 

★ Learn social and prosocial competencies 
and behaviors 

★ Build moral identity 

★ Develop prosocial and moral responsibility 

★ Develop academic interest, skills and 
performance 

Goals for Teachers 

★ Foster social and emotional self-regulation in 
students 

★ Foster and model prosocial moral responsibility 

★ Foster school and classroom community 

★ Foster and model active citizenship 

★ Foster attachment to school 

★ Foster and model engagement in learning 

★ Foster academic skills, including good study 
habits, and support academic performance 

★ Promote and model avoidance of risky behaviors 



Goals for Schools and Administrators 

★ Provide a safe and caring environment for all 

★ Promote a positive school climate and 
school culture^ 

★ Coordinate education of some or all teachers in 
knowledge and skills needed to implement the 
proposed character education program 

★ Promote inclusion and friendships between 
students in special education and those in 
regular education 

★ Decrease disciplinary problems 

★ Motivate and enable parent involvement 

★ Encourage and enable community involvement 

★ Foster values-based class discussions 

★ Encourage students’ character development 

★ Encourage students to learn and demonstrate 
prosocial and moral attitudes, behaviors, and 
competencies listed under student goals 

★ Promote the fullest social and academic inclu- 
sion of students with special needs 

★ Promote student-centered teaching and 
learning activities 

★ Encourage students’ positive academic habits 
and performance 

Goals for Parents and Community 

★ Become involved in schools and school life 

★ Provide support and encouragement to children 
and youth for character development 

★ Be role models for children and youth 

★ Provide support for character education pro- 
grams and interventions 

★ Offer and support continuation activities (e.g., 
after-school, faith-based and community pro- 
grams) for further character development 

of students 

In the evaluation plan, each goal will be turned into 
a measurable outcome. Measurements of outcomes can be 
done by using one or a combination of techniques: 
observations, questionnaires, surveys, tests, teacher re- 
ports, parent reports, and school records. 



3. For an overview of school climate and school culture, see appendix B. 



Program Requirements Area: Know the program 
requirements and features. The program description 
should describe in detail what the intervention requires 
and what is included in it. Particular features of the in- 
tervention (i.e., curriculum, activities, rules for behavior), 
who will be responsible for them (e.g., specially trained 
teachers, all teachers, student leaders, all students, admin- 
istrators, school support staff members, outside special- 
ists), and where they will take place (e.g., in classrooms, 
schoolwide, after-school programs, parent-community 
meetings) should be included. Fidelity to the planned 
program, as well as the frequency and intensity of the in- 
tervention activities should be monitored and recorded by 
the project director and evaluator team during implemen- 
tation. Possible interventions might include the following: 

★ Professional development of teachers and admin- 
istrators, including training in intervention 
techniques, strategies and goals 

★ Curricular changes; integration of character, 
moral and values-based content into existing 
curriculum 

★ Introduction of a new curriculum 

★ Integration into existing curriculum of activities 
to promote prosocial attitudes and skills 

★ New teaching techniques and strategies or 
changes in existing ones 

★ Schoolwide activities 

★ Classroom activities 

★ Partnerships with other programs 

★ Parent education and activities 

★ Changes in the organizational structure of the 
school or classroom 

★ Efforts to involve all students in school activities, 
including students with special needs 

★ Service learning curriculum 

★ Community education 

Broad Characteristics Area: Incorporate school, 
district, and community characteristics. The program 
description should discuss how the program fits with, and 
takes into account, particular characteristics of not only 
the school or district but also the community in which it 
will be implemented. Specifically, it should include clear 
descriptions of the implementation sites; their capacity to 
implement the program; and germane characteristics of 
the school, district and community. The fit of an inter- 
vention or program to a particular school or district is 



important; therefore, as many as possible of the following 
aspects should be addressed: 

★ Urban, suburban, rural 

★ Existence in the school, the district or both of 
after-school programs, wraparound services and 
so forth 

★ Extent of participation in the free and reduced 
price school meals program and other govern- 
ment subsidized school programs 

★ Extent of participation in Honors, Advanced 
Placement and other high-achievement 
programs 

★ Percentage of student body with special needs 

★ Adequacy and prominence of programs to serve 
students with special needs 

★ Percentage of student body using English as a 
second language 

★ Adequacy and prominence of remedial and 
advanced language programs 

★ Prominence of sports and clubs 

★ Visible community support for the school, the 
district or both 

★ Existence of other character or emotional de- 
velopment, social skills, and leadership training 
programs in the school or district 

Intervention Guidelines Area: Understand local, 
state and federal guidelines relevant to the interven- 
tion. In addition to including characteristics of school 
and community contexts (Broad Characteristics Area), the 
proposal narrative should address guidelines and standards 
from state and local education agencies; school boards 
and advisory groups; parent and community voices; and 
federal, state or private funding sources (e.g., the guide- 
lines for Partnerships in Character Education Program 
grants) that are pertinent to the intervention or program. 
The project director should understand the guidelines and 
standards from all levels and include references to them 
in the proposal narrative. The following are examples of 
the various intervention guidelines; specific guidelines and 
standards will vary for each grant application: 

★ Federal initiatives and guidelines, for example, 
the PCEP guidelines 

★ State initiatives and guidelines 

★ Community standards 
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★ School district guidelines that affect implemen- 
tation of the intervention 

★ School guidelines that affect implementation of 
the intervention 

SHARING THE PROGRAM DESCRIPTION 
WITH STAKEHOLDERS 

While the program description is being developed, 
the project director should seek the opinions and views of 
key stakeholders in the school system and in the com- 
munity Their ideas and perspectives can he crucial in 
presenting the strongest and clearest picture of the pro- 
gram and how it will serve not only the needs of students 
and parents but also the aspirations of the schools and the 
community. Once the program description narrative is 
complete, it is beneficial to present it to wider groups of 
stakeholders to inform them and to garner their support. 

TRANSLATING THE PROGRAM 
DESCRIPTION INTO A PROGRAM THEORY 
OE CHANGE AND LOGIC MODEL 

The following chapter. Step 3, discusses how the 
program description is used by the evaluator to create an 
evaluation plan. Sometimes evaluators will translate the 
program description into a program theory of change and 
logic model as a preliminary step to writing an evalua- 
tion plan. Writing a program theory and creating a logic 
model are becoming more common tasks of evalua- 
tors, and some evaluators find those steps to be helpful. 
However, because writing a clear program description is 
fundamental and most important, this chapter has been 
devoted to that topic. Writing a program theory and 
creating a logic model are at the discretion of each evalua- 
tor, so this chapter does not discuss those topics in depth. 
Nevertheless, resources offered at the end of this chapter 
give more in-depth ideas about what a program theory of 
change is and what purposes can be served by a good logic 
model. In short, however, a program theory of change is a 
statement of the assumptions about why the intervention 
should affect the outcomes it is expected to produce. An 
accompanying logic model would depict a figure showing 
the relationships between the program requirements and 
features and the expected outcomes. The five program 
description areas discussed in this chapter would be used 
to articulate a program theory of change and logic model. 



SUMMARY 

Writing a clear and comprehensive description of 
the program that emphasizes the program’s strengths in 
the five areas described above is the second important 
step in creating a strong grant application proposal. Each 
program and proposal will be different, thus not all of 
the points in each area will pertain to any one program 
description. The program description is important in its 
own right because it sets out an intervention’s parameters 
and goals. The program description as well as a program 
theory and logic model, if you decide to use them, serve 
as guides for the development of the evaluation plan that 
is discussed in the next chapter. Step 3. 



RESOURCES EOR PROGRAM THEORIES 
OF CHANGE AND LOGIC MODELS 



Publications 

Chen, H. 2005. Practical program evaluation: Assessing and improving 
planning, implementation and effectiveness. Thousand Oaks, Calif.: 
Sage. (See especially pages 12—44.) 

Cohen, J. 2006. Social, emotional, ethical, and academic education: 
Creating a climate for learning, participating in democracy, and well- 
being. Harvard Educational Review 76 (2): 201—37. 

Kuperminc, G.P., B.J. Leadbeater, C. Emmons, C., and S.J. Blatt. 
1997. Perceived school climate and difficulties in the social adjust- 
ment of middle school students. Applied Developmental Science 1 (2): 
76-88. 

Internet Resources 

Enhancing Program Performance With Logic Models — K course to 
help program practitioners use and apply logic models. See http: 
//www. uwex.edu/ces/lmcourse/. 

W. K. Kellogg Foundation — K tool kit on program evaluation 
targeted primarily to those W. K. Kellogg grantees working with 
outside evaluators, but of potential use to anyone seeking to design 
an effective, useful evaluation. See http://www.wkkforg/pubs/tools 
/evaluation/pub3669.pdf 




STEP 3 

PREPARE THE EVALUATION PLAN 

Step 3 focuses on how to prepare the evaluation 
plan using the program description discussed in Step 2. 
This third step includes formulating evaluation research 
questions and deciding on the most effective evaluation 
design (both process and outcome) and procedures. The 
discussion of research evaluation questions will assist 
project directors and evaluators in making the decision to 
use either an experimental or quasi-experimental design or 
another approved design. 

COLLABORATING TO DEVELOP 
THE EVALUATION PLAN 

Conducting a scientifically rigorous evaluation of 
a character education program or intervention requires 
planning and continuous communication between the 
project director and evaluator. The overall evaluation plan, 
including program assumptions and goals, research ques- 
tions, study design, and procedures for conducting the 
evaluation, is developed and written into the proposal ap- 
plication. Some of the details about evaluation procedures 
might not be feasible to decide before the grant award 
is made; however, they should be determined as soon as 
possible afterward so baseline data can be collected before 
the implementation begins. Step 2 discussed how the 
team’s fundamental understanding of the program and 
its assumptions about expected outcomes are spelled out 
in the program description. Step 3 looks more closely at 
developing the evaluation questions to be addressed, pos- 
sible research designs, and the procedures for conducting 
the evaluation. 

The written evaluation plan, including its research 
questions, research design and procedures, should be 
shared with key stakeholders, just as the program descrip- 
tion described in Step 2 was shared. Including all stake- 
holders’ perspectives, especially those of school personnel 
from both potential intervention and control or compari- 
son schools, will increase the credibility of the evaluation 
plan and will contribute to a more valid evaluation. 

Designing the evaluation plan should be a collabora- 
tive effort. The project director and the evaluator as the 
key team members should pool their expertise about not 
only what will enable the evaluation but also what may 
limit or obstruct it. These discussions require an invest- 
ment of time for assessing details, deliberating and 
building consensus. 



The evaluation questions and the evaluation design 
are directly informed by the program description. As 
mentioned in Step 2, the program description spells out 
the goals and processes of the intervention or program; 
thus, they serve as a guide for generating the evaluation 
research questions. 

WRITING EVALUATION QUESTIONS 

The evaluation questions propose what various us- 
ers and stakeholders need and want to know about the 
intervention. Initially, the project director and the evalu- 
ator will benefit from discussing the following questions 
that relate to the Context Area described in Step 2. That 
discussion will help to generate a useful foundation for 
the research project. 

★ What does existing research tell us about effec- 
tive character education interventions? 

★ What are the most important elements of those 
interventions? 

★ What do you think are the most important 
elements of your intervention? 

★ How many of your most important elements are 
the same as or similar to those in the effective 
interventions found in the existing research 
described in the first question? 

★ How did the schools, students, teachers, families 
and community change as a result of the 
interventions found in the existing research 
mentioned above? 

★ How should schools, students, teachers, families 
and communities change as a result of your 
intervention? 

★ What determines the extent of the effective- 
ness of the character education interventions 
reviewed? 

Once the project director and evaluator have 
discussed the above questions, they are ready to focus on 
more specific research questions for their own project. 
Formulating specific research evaluation questions will 
generate ideas about the kind of information that is 
needed to address each question, how that information 
will be gathered, and how it will be analyzed to most 
directly answer each question. The detailed descriptions of 
the chosen program’s goals and features given in response 
to the Goals Area and the Program Requirements Area, 
discussed in Step 2, will provide the information necessary 
to write appropriate, clear and precise research questions. 
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EXHIBIT 2 

MODEL FOR EVALUATION QUESTIONS WORKSHEET 


EVALUATION 

RESEARCH 

QUESTION 


PURPOSE OF THE 
QUESTION— WHAT 
SHOULD THE 
ANSWER 
DEMONSTRATE? 


WHAT INFORMATION 
WILL BE NEEDED 
TO ANSWER THE 
QUESTION? 


WHEN AND 
HOW WILL THE 
INFORMATION BE 
COLLECTED? 


HOW WILL THE 
DATA BE ANALYZED 
TO BEST ANSWER 
THE QUESTION? 




















































Source: Adapted from Sanders, 2000. 



In addition, the worksheet model shown in Exhibit 2 can 
he used for structuring initial discussions between the 
project director and evaluator. Using this model, or one 
similar to it, will help them formulate questions that will 
most effectively evaluate the intervention and respond 
to the grant application guidelines. In the process, it is 
also important to obtain input from each member of the 
collaborative advisory team. Different projects will have 
varying numbers of evaluation research questions. After 
writing the research questions with the project director 
and with input from stakeholders, the evaluator will be 
prepared to decide on the study design. 

UNDERSTANDING PROCESS AND 
OUTCOME EVALUATIONS 

In developing an evaluation design, it is important 
to remember that there are two aspects of an intervention 
that need to be evaluated; the processes and the out- 
comes. Exhibits 3, 4 and 5 display key characteristics and 
examples for evaluating processes and outcomes. 

The process evaluation, sometimes known as forma- 
tive evaluation, is designed to provide information with 
respect to (a) the fidelity of the implementation to the 



strategies planned and (b) the frequency and intensity of 
the various activities. It involves collecting, compiling and 
analyzing information related to program implementa- 
tion. The process evaluation is based on the descriptions 
given in response to the Program Requirements Area 
discussed in Step 2, and the results describe how well 
the intervention was implemented. These results can be 
used for accountability purposes. Most important, a good 
process evaluation is the foundation for the outcome 
evaluation. 

The outcome evaluation study is designed to deter- 
mine whether an intervention produced the expected or 
intended effects. In other words, it determines whether 
and how well a program met its goals as delineated in the 
Goals Area; thus, the outcomes study provides impor- 
tant data on how effective the program is as a character 
education intervention. Outcome evaluations involve (a) 
collecting data about the districts and schools themselves 
and (b) using appropriate instruments to collect data from 
students, teachers, administrators, parents and com- 
munity members with respect to specified intervention 
outcomes (the Goals Area). 

In summary, the process evaluation determines how 
well an intervention is put into place, how well it delivers 




EXHIBIT 3 

KEY CHARACTERISTICS OF PROCESS AND OUTCOME EVALUATIONS 



CHARACTERISTIC 


PROCESS EVALUATION 


OUTCOME EVALUATION 


PURPOSE 


To determine implementation fidelity 
(the extent to which intervention 
strategies and activities are done as 
planned, including adherence to 
schedules) 

To determine the frequency and 
intensity of the intervention activities 

To determine the extent to which the 
delivery of the intervention was achieved 

May be used to provide feedback to 
improve an intervention 


To determine the extent to which the intervention 
as implemented achieved its intended goals and 
addressed the issues and needs it was intended to 
address 


DESIGN 


The process evaluation is designed to 
measure intervention implementation 
processes. Process evaluation begins 
at program inception and continues 
at varying rates throughout an 
intervention’s lifecycle. 


The outcome evaluation is designed to determine 
whether the intervention has met its purpose and 
goals. Several important design issues must be 
considered, including how to best determine the 
results and how to best contrast what happens as 
a result of the intervention with what happens 
without the program. 

Experimental designs use a combination of 
experimental groups and control groups to 
obtain the highest quality scientific answer to the 
question of outcome. 

Quasi-experimental designs are used when it is 
impossible to use experimental designs and when 
some comparison is needed. 


REPORTING 


Process evaluation findings are reported 
in lay language to all stakeholders, 
including the school community and 
funding agencies. 


Outcome evaluation findings are reported 
as scientific research to SEAs, LEAs, and the 
professional and research communities through 
professional presentations, journals, and books as 
well as in reports to funding agencies, the school 
community, and other stakeholders. 


USE OF 
FINDINGS 


Findings cannot be generalized to future 
use of the intervention. Findings can be 
used to define and set new standards for 
the present intervention. 


Findings can be used to support using the 
intervention in other school systems, while being 
sensitive to contextual differences and necessary 
adaptations. 



its services, and how well it maintains fidelity to the pro- 
gram as designed. Then the outcome evaluation assesses 
an intervention’s effectiveness in achieving its goals for 
positively affecting stakeholder groups, including stu- 
dents, teachers, schools, parents and the community. 

Exhibits 3, 4 and 5 provide more details about pro- 
cess and outcome evaluations. Exhibit 3 presents four key 
characteristics of both process and outcome evaluations — 
the purpose, the research design to address the purpose. 



how the evaluation is reported and to whom, and how its 
findings can be used. 

Exhibit 4 presents examples of research questions, 
methods, and the value of the results that should be con- 
sidered when designing a process evaluation. 

Exhibit 5 presents examples of questions, methods, 
and the value of the results for use when designing an 
outcome evaluation. 
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EXHIBIT 4 

SAMPLE QUESTIONS, METHODS AND VALUE OF RESULTS FOR PROCESS EVALUATIONS 


QUESTION 


METHOD 


VALUE OF RESULTS 


To what extent was the 
intervention implemented 
as designed? 


Compare the amount and range of 
activities done in the intervention 
with that prescribed by the program 
developers. 


The comparison gives an indica- 
tion of the fidelity of the imple- 
mentation to the planned program 
and the frequency and intensity of 
the intervention activities. 


What adaptations, additions and 
omissions were made when the 
intervention was implemented? 


Record, describe and count. 


Adaptations, additions and 
omissions affect the analyzing of 
data for the outcome evaluation. 


To what extent were the 
character educators (e.g., 
teachers) trained? 


Compare with the standards of 
optimal training as prescribed by the 
program developers. 


The comparison gives an 
indication of the potential strength 
or weakness of the intervention. 


To what extent are stakeholders 
informed and knowledgeable 
about the intervention? 


Maintain records of meetings and 
presentations to stakeholders as well 
as questionnaire responses from 
stakeholders. 


The information gives an 
indication of the range of 
stakeholders and their knowledge. 



Although the content of these examples may be 
useful, each intervention has its own overall purpose and 
specific goals, and each evaluation project should cap- 
ture the specific features related to it. The designs of the 
process and outcome evaluations should take into account 
the specific aspects of the intervention as spelled out in 
the program description developed in Step 2, with atten- 
tion to local, state and federal guidelines relevant to the 
intervention. 

Because an evaluation design focuses on intentional 
interventions, measurable outcomes, and procedures for 
measuring outcomes, the design determines not only 
what data will be collected but also what procedures will 
be used for data collection and analysis. The program 
description and evaluation research questions are the 
foundations for developing an evaluation plan, especially 
the research design. 

UNDERSTANDING EXPERIMENTAL AND 
QUASI-EXPERIMENTAL RESEARCH DESIGNS 

In an experimental research design, also known as a 
randomized controlled trial, outcomes are monitored and 
measured for two similar groups, called samples: (a) the 
intervention, or experimental, group and (b) the control, 
or nonexperimental, group. The participants who make 
up the groups are usually selected from a pool of potential 
individuals, classrooms or schools who have volunteered 
to receive services or to participate in an intervention or 



program. These individuals, classrooms or schools are 
then randomly assigned to either the experimental or 
control group. 

Because the individuals, classrooms or schools are 
randomly selected in the exact same way to participate in 
one of the two groups, any differences between the groups 
should exist only by chance. All known (i.e., measur- 
able) and unknown (i.e., not measurable) factors should 
be represented to the same degree in both groups. The 
unique advantage of random assignment is that it makes 
it possible for the evaluation process to isolate and deter- 
mine whether the intervention itself caused the intended 
outcomes, with no other explanations being possible. The 
following example was offered in the U.S. Department 
of Education’s publication. Identifying and Implementing 
Educational Practices Supported by Rigorous Evidence: A 
User-Eriendly Guide: 

[You] want to test, in a randomized controlled trial, 
whether a new math curriculum for third-graders 
is more effective than your school’s existing math 
curriculum for third-graders. You would randomly 
assign a large number of third-grade students to 
either an intervention group, which uses the new 
curriculum, or to a control group, which uses the ex- 
isting curriculum. You would then measure the math 
achievement of both groups over time. The difference 
in math achievement between the two groups would 
represent the effect of the new curriculum compared 
to the existing curriculum. (USED/IES 2003, 1). 




EXHIBIT 5 

SAMPLE QUESTIONS, METHODS AND VALUE OF RESULTS FOR AN OUTCOME EVALUATION 



QUESTION 


METHOD 

(USED BEFORE AND AFTER INTERVENTION 
ON TWO OR MORE OCCASIONS) 


VALUE OF RESULTS 


Does the intervention affect 
school culture and targeted 
aspects of school climate? 


Compare the change in school culture and 
targeted aspects of climate of intervention 
schools with the school climate of control or 
comparison schools. 


Positive findings suggest that the 
intervention may be a source of 
positive school culture and aspects 
of climate. 


Does the intervention positively 
affect school culture and targeted 
aspects of school climate? 

Do school culture and the 
targeted aspects of school climate 
affect student outcomes? 


Use methods that assess (a) school culture 
and targeted aspects of school climate and (b) 
whether changes in the culture and climate of 
intervention schools are necessary or helpful in 
promoting positive student outcomes. 


Positive findings suggest that the 
intervention changes school culture 
and targeted aspects of school 
climate and that those changes 
promote positive student outcomes. 


Does the intervention promote 
higher levels of moral and value- 
based reasoning? 


Compare intervention students’ levels of 
moral and values-based reasoning with those 
of students in control or comparison groups, 
taking into account other aspects of the schools. 


Positive findings suggest that the 
intervention promotes students’ 
moral and values-based reasoning 
abilities. 


Does the intervention promote 
more social and prosocial 
competencies and behaviors by 
students? 


Compare intervention students’ levels of social 
and prosocial competencies and behaviors 
with that of students in control or comparison 
groups, taking into account other aspects of the 
schools. 


Positive findings suggest that the 
intervention increases students’ 
social and prosocial competencies 
and behaviors. 


Does the intervention 
promote fewer and less serious 
incidents requiring referrals 
to administrative offices for 
discipline? 


Compare the occurrence of referrals to an 
administrator’s office for intervention students 
with those of students in control or comparison 
groups, taking into account other aspects of the 
schools. 


Positive findings suggest that 
the intervention results in fewer 
referrals to an administrator’s office 
for disciplinary problems. 


Does the intervention promote 
students’ attachment to school 
and academic achievement (e.g., 
grades, test scores, portfolios and 
other school assignments) ? 


Compare intervention students’ feelings of 
school attachment and academic achievement 
with those of students in control or comparison 
groups, taking into account other aspects of the 
schools. 


Positive findings suggest that the 
intervention enhances students’ 
feelings of attachment to school 
and promotes their academic 
achievement. 


Do students perceive their 
schools as safe and caring? 


Compare intervention students’ perceptions of 
school safety and caring with those of students 
in control or comparison groups, taking into 
account other aspects of the schools. 


Positive findings suggest that the 
intervention enhances students’ 
sense of being safe and cared for at 
school. 


Does the intervention increase 
teachers’ use of student-centered 
pedagogies and learning 
activities? 


Compare intervention teachers on their use 
of student-centered pedagogies and activities 
with that of teachers in control or comparison 
groups. 


Positive findings suggest that the 
intervention changes the way 
teachers teach. 


To what extent are parents (or 
other stakeholders such as school 
administrators, school support 
staff members, community mem- 
bers, local businesses and local 
community agencies) involved in 
school and school life? 


Compare the extent to which parents with 
children in intervention schools are involved in 
their schools with the extent to which parents 
whose children are in the control or comparison 
groups are involved in their schools and school 
life. 


Positive findings suggest that the 
intervention affects parent (or other 
stakeholders’) involvement in school 
and school life. 
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In a variation on this basic concept, participants may 
be randomly assigned to a control group and to two or 
more different intervention groups, which enables one 
study to measure the effects of different interventions 
that target the same outcomes. For instance, a number of 
schools that use Character Education Intervention A might 
be measured against other schools that use Character Edu- 
cation Intervention B, and both of those groups might be 
measured against one or more control groups that did not 
implement any type of character education program. 

Quasi-experimental design uses nonrandom pro- 
cedures to assign students, classrooms or schools to 
intervention groups and to comparison groups that are 
assumed to comprise the same factors influencing the 
outcomes. Quasi-experimental design studies compare 
outcomes for those taking part in the intervention with 
outcomes for those in comparison groups. All groups are 
chosen through some method other than randomization. 

Intervention and comparison groups are typically 
matched on a variety of characteristics. Schools are often 
matched by size, by grades included in the school, and by 
teacher and student composition. Classrooms are often 
matched by subject, learning track, and so forth. Students 
are usually matched according to background character- 
istics such as age, gender, and ethnicity as well as by mea- 
sures of learning such as test scores and learning tracks. 
However, even groups that are well matched on criteria 
like these may still be very different with respect to other 
characteristics that may have an independent effect on the 
outcomes, especially in character education. Data from 
quasi-experimental studies are analyzed using statistical 
techniques that adjust for these other characteristics that 
(a) are found to be different between the two groups at 
the beginning of the evaluaton (its baseline) and (b) may 
independently explain any of the outcomes of interest. 

Findings from nonexperimental studies such as 
quasi-experimental designs should be considered sugges- 
tive. There are always unmeasured factors or variables that 
cannot be studied. In other words, outcomes cannot be 
claimed to result exclusively from the intervention because 
they could be attributed to other considerations that either 
were outside the scope of the study or were never mea- 
sured. Nevertheless, using some comparison groups that 
make an effort to match the program group is better than 
either no comparison or a simple pre-post study design. 



For studies in the education field, intervention and 
comparison groups are often matched closely on charac- 
teristics such as the following: 

★ Prior test scores and other measures of academic 
achievement, prosocial attitudes and behaviors, 
and moral and values-based reasoning abilities — 
preferably, the same measures that the study will 
use to evaluate outcomes for the two groups 

★ Demographic characteristics such as age, sex, 
ethnicity, poverty level, parents’ educational 
attainment, and single- or two-parent family 
background 

★ Time period in which the two groups are studied 

★ Methods used to collect outcome data (e.g., the 
same test of reading skills administered in the 
same way to both groups) (USED/IES 2003) 

Exhibit 6 presents key characteristics of experimental 
and quasi-experimental designs, their cost, as well as their 
advantages and disadvantages. 



In 2002, the Department of Education strengthened 
the priority for outcome evaluations for the Partnerships 
in Character Education Program as well as for many 
other programs. The Department expressed interest in 
evaluations that use rigorous, scientifically based research 
methods to assess the effect of character education inter- 
ventions on student achievement or teacher performance. 
Both experimental designs with randomly assigned groups 
and quasi-experimental designs with carefully matched 
comparison groups were encouraged. It is important to 
note, however, that in 2004, the Department of Educa- 
tion broadened, in some circumstances, tbe types of 
evaluation research designs that could be considered 
scientifically based to include regression discontinuity and 
single case-study designs (USED/OSDFS 2004). For the 
purpose of this document, we will focus only on experi- 
mental and quasi-experimental designs. 



EXHIBIT 6 

EVALUATION DESIGN CHARACTERISTICS 



DESIGN 


CHARACTERISTICS 


PERCENTAGE 
OF OVERALL 
PROJECT 
BUDGET^ 


ADVANTAGES 


DISADVANTAGES 


EXPERIMENTAL 

DESIGN 


Incorporates random assign- 
ment of participants to inter- 
vention and control groups. The 
purpose of randomization is to 
ensure that all possible explana- 
tions for changes (measured and 
unmeasurable) in outcomes are 
taken into account, randomly 
distributing participants in both 
the intervention and control 
groups so there should be no 
systematic baseline differences. 
Intervention and control groups 
are compared on outcome 
measures. Any differences in 
outcomes may be assumed to be 
attributable to the intervention. 


35-55 

percent'’ 


Most sound 
or valid 
study design 
available 

Most 

accepted in 

scientific 

community 


Institutional policy 
guidelines may make random 
assignment impossible. 


QUASI- 

EXPERIMENTAL 

DESIGN 


Involves developing an 
intervention group and a 
carefully matched comparison 
group (or groups). Differences 
in outcomes between the 
intervention and comparison 
groups are analyzed, controlling 
for baseline differences 
between them on background 
characteristics and variables of 
interest. 


35-55 

percent'’ 


More 
practical 
in most 
educational 
settings 

Widely 
accepted in 
scientific 
community 


Finding and choosing suitable 
intervention and comparison 
groups can be difficult. 

Because of nonrandom group 
assignment, the outcomes of 
interest in the study may have 
been influenced not only by 
the intervention but also by 
variables not studied. 


a. An evaluation budget may include, but not be limited to, the following: evaluator’s fee, costs associated with acquiring parental consent, cost of 
IRB review, cost of printing and mailing surveys, cost of hiring and training data collectors. 

b. The percentage of funds allocated for evaluation depends on the research design and the scale of the project. Large projects that include several 
sites (e.g., school districts) with many schools and thousands of participants will need a lower percentage of the overall budget than small-sized 
projects in one school district. 
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DECIDING SAMPLE SIZE 

The entity — the student, classroom or school — 
that is to become the study’s sample is called the unit of 
analysis. Because most character education interventions 
involve whole-school activities and school climate change, 
the entity studied may often be the school or the class- 
room. Ideally, an evaluation design should include an ap- 
propriate number of these units so results are meaningful. 
The number needed to detect a desired or expected dif- 
ference in effects between the intervention and control or 
comparison groups can be determined through a technical 
statistical procedure called power analysis. The project 
evaluator should conduct an appropriate power analysis 
after determining whether the units will be students, class- 
rooms or schools. Power analyses should be done during 
proposal development and should be included in the 
grant application narrative. A helpful resource is Applied 
Multiple Regression/ Correlation Analysis for the Behavioral 
Sciences (Cohen et al. 2003). Another helpful resource is 
a recent paper “Statistical Power for Random Assignment 
Evaluations of Education Programs” (USED/IES 2005). 

Randomly selecting or assigning the appropriate 
number of schools to intervention and control or com- 
parison groups can make character education evaluation 
a challenge. The project director and evaluator should 
develop a plan for recruiting and retaining the sample of 
schools, classrooms or students throughout the study. It is 
important that the evaluator specify the sample character- 
istics and sample size and that the project director advise 
the evaluator about the feasibility of obtaining the desired 
number of schools or other units of analysis for the 
sample. A fully adequate sample, including one that is suf- 
ficiently large, is critical if the evaluation is to yield valid 
and reliable conclusions about the effects of the character 
education program. 

RECOGNIZING THREATS TO VALIDITY 

Project directors should be aware of the common 
issues that can threaten the validity of an evaluation. A 
valid evaluation study is one that uses sound measures, 
analyzes the data correctly for the design, and bases its 
inferences on the study’s findings. Threats to the validity 
of a research evaluation may include the following;'* 

★ Poor implementation of the intervention or 
lack of intervention fidelity — If the imple- 



mentation deviates from the intervention or 
program as designed by the developers, then 
the validity of the outcome evaluation will be 
compromised. Determining intervention fidel- 
ity is a key element of the process evaluation as 
described earlier in this chapter. 

★ Subject selection bias — Intervention partici- 
pants might differ from comparison participants 
in important ways that may affect the ability to 
detect the intended intervention effect. Includ- 
ing participants who volunteer to be a part of 
the evaluation or who are specifically targeted 
for participation may make it difficult to find 
comparable comparison groups. Consequently, 
selected participants in the intervention and 
comparison groups might not be matched in a 
balanced way; for example, one group may be 
more involved in school and after-school activi- 
ties to begin with. 

★ Subject attrition — Participants may drop out 
of the study or move out of the school or school 
district. 

★ Differential history of participants — Par- 
ticipants may have different backgrounds that 
influence attitudes, competencies, and behaviors. 
This variation is less of a concern in experimen- 
tal design because of the random assignment of 
the intervention and control groups. 

★ Problems with outcome measures, including 
poor validity, poor reliability or instrument 
reactivity — Tools or surveys used for measuring 
outcomes of interest may not meet acceptable 
scientific standards, for example, instruments 
may not be field-tested before use in the study. 
Such problems could lead to design breakdown. 

The evaluator should discuss the above issues with 
the project director during the planning process and 
should specify procedures to minimize the likelihood of 
these threats occurring. The best protection against design 
breakdown depends on well-planned and well-implement- 
ed interventions and evaluations that are executed in part- 
nership by an informed and committed project director 
and evaluator, both of whom are supported by adequate 
time and resources. 



4. You will find more discussion about possible threats to the validity of 
an evaluation in Step 7, Monitoring for Issues in Data Analysis. 



DEVELOPING DATA COLLECTION 
PLANS AND PROCEDURES 

The next component of the evaluation is the plan for 
data collection. The plan must specify the data needed 
and the data collection procedures to be used for each 
outcome identified in the program description. Although 
the evaluator should guide the development of the data 
collection plan — which must receive Institutional Review 
Board (IRB) approval (see Step 4) — the project director 
and school representatives should contribute substantially 
to the plan to ensure that data are gathered in a structured 
and systematic fashion that causes the least disruption 
to the schools’ daily operations. Outlining the data 
collection procedures in advance is necessary to identify 
logistical problems; pinpoint how to ensure the collection 
of essential data; and in some cases, determine whether 
it is necessary to change the kind or amount of data col- 
lected. The data collection plan should include procedures 
to do the following: 

★ Specify the data needed initially for the baseline 
and periodically throughout the time the inter- 
vention is evaluated 

★ Obtain data to verify which and to what extent 
the purpose and goals were met and outcomes 
were achieved 

★ Identify data sources (see Exhibit 7) 

★ Design or obtain instruments or other means of 
collecting data (parent or teacher surveys, school 
attendance records, discipline referral forms, 
classroom observations, log sheets for parent 
contacts, activity logs, and interview formats) 

★ Ensure that obtained instruments are valid 
(that they measure what they are supposed to 
measure), are reliable (that they measure what 
they measure in the same way each time they are 
used), and are developmentally and culturally 
appropriate 

★ Ensure that measures developed for the evalua- 
tion (a) are analyzed for validity and reliability 
on the samples in the study and (b) are devel- 
opmentally and culturally appropriate for the 
samples in the study 

★ Secure (a) full parental consent and child assent 
for all evaluation-related procedures and (b) 
consent from all other adults in samples 

★ Schedule data collection to create the least 
conflict with the school calendar and to occur 



at appropriate intervals (before implementation, 
periodically during the implementation, imme- 
diately after the implementation, and six months 
after the implementation if it has an end point) 

★ Specify who will collect data (When possible, 
independent data collectors should be hired by 
the evaluator; when impossible, the evaluator 
may, in some cases, use staff members involved 
in the intervention, school personnel, or a 
combination.) 

★ Administer instruments efficiently, being 
mindful of the length of time needed, materi- 
als required, training requirements, and type of 
administration (group versus individual) 

★ Develop a data collection manual and training 
program or other means to collect data in ways 
that will ensure their validity 

★ Score, manage and analyze data 



Exhibit 7 provides data sources often used to collect 
the kinds of data usually assessed in evaluations of charac- 
ter education interventions and programs. 




Exhibits 8 and 9 present examples of data collection 
matrices for process and outcome evaluations. Each ma- 
trix is incomplete, giving only a few examples of program 
components (Exhibit 8) and a few examples of measurable 
outcomes (Exhibit 9). 
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EXHIBIT 8 

DATA COLLECTION MATRIX FOR PROCESS EVALUATIONS 


EXAMPLES 
OF PROGRAM 
COMPONENTS 


DATA ELEMENTS 


DATA SOURCES 


MEANS OF 
COLLECTING DATA 


WHEN 

COLLECTED 


Character 
education 
integrated into 
the curriculum 


List of traits, 
values or 
specific program 
content and 
activities 

Integration into 
lesson plans 

Integration 
into teaching 
strategies 


Curricula 

Project director’s 
observation notes 

Lesson plans 

Intervention staff 
members 

Teachers 

Administrators 


Teacher and adminis- 
trator interviews 

Teacher self-reports 


Throughout the 
year 


Community 

partnerships 


List of traits, 
values or 
specific program 
content and 
activities 

Character 
education 
priorities, 
mission or 
policy statement 

Meeting dates, 
times and 
content 


Meeting attendance 
rosters 

Meeting minutes 

Action or strategic 
plans 

Parents 

Community members 


Surveys 

Interviews 

Observation protocols 
Focus groups 


Mid-year and 
end of year 


Communication 


Meeting dates, 
times and 
content 


Project director 
Implementers 
School administrators 


Project records of 
content activity 

Project director’s 
feedback 

Evaluator’s feedback 


Mid-year and 
end of year 




EXHIBIT 9 

DATA COLLECTION MATRIX FOR OUTCOME EVALUATIONS 


EXAMPLES OF 
MEASURABLE 
OUTCOMES 


DATA ELEMENTS 


DATA SOURCES 


MEANS OF 
COLLECTING DATA 


WHEN COLLECTED 


Particular level 
of safety and 
caring in a 
school 


Student, parent, 
teacher, administrator 
perceptions of school 
safety and school as a 
caring community 


Students, 
their parents, 
teachers, and 
administrators 


Surveys of 
students, parents, 
teachers, and 
administrators 


Early spring each 
year 


Decreased 
number and 
severity of 
incidents 
requiring 
referrals to 
administrative 
offices for 
discipline. 


Acts against persons 

Acts against property 

Failure to comply with 
rules 

Possession of drugs or 
weapons 


School records 


Student referral 
tracking form 


Periodically, with 
periods to be 
decided based on 
evaluation purposes 


Improved 
student 
prosocial 
attitudes and 
behaviors, 
moral and 
values-based 
reasoning, social 
and emotional 
competencies 


Curriculum 
activities, use 
of conflict resolution 
strategies, classroom 
discussions, 
community service 


All students 


Student surveys, 
observations, 
teacher reports, 
parent reports 


Periodically, with 
periods to be 
decided based on 
evaluation purposes 


Improved levels 
of achievement 
in reading, math 
and writing 


Reading scores, math 
scores and writing 
scores on standardized 
tests 

Student course grades 


All assessed 
students 

All students 


Standardized 

tests 

School academic 
records 


Spring each year 
or whenever 
standardized test 
scores become 
available 



SUMMARY 

An effective evaluation plan is built on and guided by 
the program description. It includes research questions, a 
study design, as well as data collection and analysis proce- 
dures. It should be shared with key stakeholders to ensure 
its credibility and to garner necessary school, district and 
community support. Finally, as is true for the program 
description, the evaluation plan should be developed by 
the evaluator, working closely with the project director. 
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RESOURCES FOR DEVELOPING 
EVALUATION PLANS 



Publications 

Brand, S., R. Felner, M. Shim, A. Seitsinger, andT. Dumas. 2003. 
Middle school improvement and reform: Development and valida- 
tion of a school-level assessment of climate, cultural pluralism, and 
school safety. Journal of Educational Psychology 95 (3): 570—88. 

Connell, J., A. Kubisch, L. Schorr, and C. Weiss, eds. 1995. New ap- 
proaches to evaluating community initiatives. Vol. 1, Concepts, methods 
and contexts. Washington, D.C.: Aspen Institute Press. 

Connell, J., K. Fulbright-Anderson, and A. Kubisch 1998. New ap- 
proaches to evaluating community initiatives. Vol. 2, Theory, measure- 
ment and analysis. Washington, D.C.: Aspen Institute Press. 

Higgins-D’Alessandro, A., and D. Sadh. 1997. The dimensions and 
measurement of school culture: Understanding school culture as the 
basis for school reform. International Journal of Educational Research 
27 (7): 553-69. 

Rossi, R, H. Freeman, and M. Lipsey. 2004. Evaluation: A systematic 
approach. 7th ed. Thousand Oaks, Calif.: Sage. 

Shadish, W. R., Cook, T.D., and Campbell, D.T. 2002. Experimental 
and Quasi-experimental designs for generalized causal inference. Boston: 
Houghton-Mifflin. 

U.S. Department of Education, Institute of Education Science 
(USED/IES). 2005. Key items to get right when conducting a random- 
ized controlled trial in education. Prepared by the Coalition for 
Evidence-Based Policy, in partnership with the What Works Clear- 
inghouse. Washington, D.C.: USED/IES. See http://www 
.whatworkshelpdesk.ed.gov/guide_RCTpdf 

Van Houtte, M. 2005. Climate or culture? A plea for conceptual 
clarity in school effectiveness research. School Effectiveness and School 
Improvement 16 (1): 71—89. 

Internet Resources 

Building Capacity to Evaluate Group-Level Interventions — K source 
for Optimal Design software. See http://sitemaker.umich.edu 
/ group-based/ optimal_design_software. 

Outcome Measurement Resource Network — K Web site maintained 
by the United Way, which makes resources related to the measure- 
ment of program outcomes available to the public. See http: 
//national. unitedway.org/outcomes/library/pgmomres.cfm. 




STEP 4 

PREPARE AND OBTAIN 
INSTITUTIONAL REVIEW 
BOARD (IRB) APPROVAL 



An Institutional Review Board (IRB) is a board 
established under federal regulations (34 CFR 97) to 
approve, request modification of or disapprove research 
activities, based on compliance with federal human 
subject regulations. An IRB may be established within a 
research university, private firm, nonprofit organization or 
even a school district. Its main charge is to protect human 
participants in studies by holding organizations and 
evaluators accountable to federal regulations that 
safeguard research participants. 

If the U.S. Department of Education determines 
that a proposed project includes nonexempt human 
subjects research, then the Department will contact the 
grant applicant to request the materials needed for human 
subjects clearance. Next, grantees, including PCEP 
grantees, are required to submit information with respect 
to the proposed research plan to an IRB for review and 
approval before the evaluation can begin. The proposal 
submitted to the IRB should include evaluation protocol, 
data collection instruments, recruitment materials, con- 
sent documents and any other information that the IRB 
may require. A nonexempt human subjects research study 
also must have a federalwide assurance (FWA) to abide by 
federal regulations and an IRB approval for the particular 
study that is being proposed. Very few PCEP evaluations 
will be considered exempt from IRB review under the 
Federal Policy for the Protection of Human Subjects, or Com- 
mon Rule (34 CFR 97; also see USED/GPOS 2005b). For 
the 17 federal agencies that have adopted it, the Common 
Rule governs the use of human subjects in research. In ad- 
dition, PCEP evaluations must meet Family Educational 
and Privacy Rights Act (FERPA) requirements if student 
records are used and Protection of Pupil Rights Amendment 
(PPRA) requirements if student surveys are used (more 
information about these policies are in appendix A). 

An FWA is a pledge that the entity will abide by 
federal regulations for protection of human subjects re- 
search (34 CFR 97; also see USED/GPOS 2005b) when 
conducting nonexempt human subjects research. Because 
an entity may conduct studies funded by various federal 



agencies, the Department uses the FWA, which is good 
for research funded by many federal agencies and can be 
renewed when it expires at the end of three years. An as- 
surance applies to an organization such as a university. An 
evaluator affiliated with an organization should request to 
use its FWA. An FWA form and instructions are 
available online at http://www.hhs.gov/ohrp/. If an 
individual evaluator is not affiliated with an organization, 
then he or she can obtain an Independent Investigator 
Agreement from theDepartment of Education.^ 

Understanding and coping with the IRB process is 
new for many educators. This chapter offers an overview 
of the IRB process and criteria. (See also Step 5, which ex- 
plores obtaining the consent of participants.) The project 
director must understand these topics because the human 
subjects protection that the IRB provides is important and 
because an evaluation that includes nonexempt human 
subjects research cannot proceed without IRB clearance. 

All project team members and data collectors should 
understand the requirements for conducting ethical 
research with human participants. The National Institutes 
of Health offers a free online course in Human Participant 
Protections Education for Research Teams that many 
school-based researchers find useful (see resource list at 
the end of this chapter). In addition, many universities 
and other institutions participate in the Collaborative IRB 
Training Initiative (CITI), online training in protection 
of human subjects, which includes separate instructional 
modules for social and behavioral researchers (see resource 
list at the end of this chapter). 

Nearly all research universities, many research firms 
and nonprofit organizations, and some urban school dis- 
tricts have their own IRBs. Most often, the evaluator han- 
dles the IRB submission process. If neither the evaluator 
nor project director is affiliated with an institution such 
as a university that has its own IRB, then they can choose 
from several options such as submitting the application to 
another entity’s IRB, setting up and registering an IRB, or 
contracting for the services of a commercial IRB. 

The evaluator will need to coordinate all the informa- 
tion that is included in the application for approval that 
is submitted to the IRB. Basically, an IRB will consider 
all of the elements listed in Exhibit 1 0 when determining 
approval for an evaluation. 



5. More information about independent investigator Agreements can be 
found on the Department's Web site at http://www.ed.gov/poiicy/fund 
/guid/humansub/guidance.htmi. 
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EXHIBIT 10 
CRITERIA USED BY AN 
INSTITUTIONAL REVIEW BOARD 
TO DETERMINE APPROVAL 
FOR AN EVALUATION 



Study design: An IRB application should specify 
how participants are recruited, selected and assigned 
to groups; the reliability and validity of measures 
and data collection instruments; and the methods of 
data analysis. 

Risks and benefits: The IRB evaluates (a) whether 
the risks to participants are reasonable in relation to 
the anticipated benefits and (b) the importance of 
the knowledge reasonably expected to result from 
the evaluation research. 

Equitable selection of participants: The IRB usu- 
ally (a) considers the purpose of the research and the 
place in which data will be collected and (b) closely 
examines any proposed study involving vulnerable 
subject populations such as children, prisoners, 
people with cognitive disorders, and economically or 
educationally disadvantaged people. 

Identification of participants and confidentiality: 

The IRB reviews the researcher’s planned methods 
for identifying and contacting potential participants 
as well as for ensuring participants’ privacy and 
confidentiality. 

Qualifications: The IRB ensures that the research 
procedures are consistent with sound research design 
and with protection of human participants. In addi- 
tion, the IRB considers the adequacy of the facilities 
and equipment to be used not only in conducting 
the research but also in maintaining the rights and 
welfare of the participants. 

Consent: The process of obtaining participants’ 
consent to be included in the evaluation study goes 
to the heart of the matter of ethical research. The 
IRB often focuses a great deal of attention on the 
issue of consent. 

Source: Adapted from Fink, 2005. 



In addition to documentation to meet the criteria 
shown in Exhibit 10, the IRB application will usually 
include 

★ the advance letters that will be sent to the par- 
ticipants in the study, including teachers and the 
families of students; 

★ flyers or letters inviting people to participate in 
the study, if applicable; 

★ the criteria for including participants in the 
study; 

★ consent forms giving all study participants the 
opportunity to decide freely whether or not to 
participate (see Step 5 for more in-depth infor- 
mation about obtaining consent); and 

★ procedures for students who do not want to 
participate or whose parents do not allow their 
participation. 

The IRB committee then reviews the application 
to determine (a) whether the risks to participants are 
minimal and reasonable in relation to anticipated benefits 
and (b) whether the selection of participants is equitable. 
The IRB will take one of three actions: (1) approve the 
application, (2) return it for revision and resubmission, 
or (3) reject it outright. If the IRB does not approve the 
submission, it will state why and provide grantees with an 
opportunity to resubmit with the appropriate documenta- 
tion or procedural changes. 

Only after receiving approval notification from 
the IRB and after completing the approved participant 
consent procedures may data collection begin. The IRB 
approval is good for up to one year. If the research will 
still be under way at the approval’s expiration date, it will 
need a continuation approval from the IRB. 




RESOURCES EOR LOCATING AN IRB AND 
PROCEEDING THROUGH THE IRB PROCESS 



Publications 

Fink, A. 2005. Evaluation fundamentals. 2nd ed. Thousand Oaks, 
Calif.: Sage. 

Sherblom, S. 2004. Issues in conducting ethical research in character 
t&\xcdXion. Journal of Research in Character Education 1 (2): 107—28. 

Internet Resources 

Collaborative IRB Training Initiative — Online training in protection 
of human subjects. See http://www.citiprogram.org/citi_information 
.asp. 

Office of Human Research Protections (OHRP), National Institutes 
of Health (NIH) — A database of registered IRBs, searchable by loca- 
tion, is available online. OHRP also provides information on federal- 
wide assurances. See http://ohrp.cit.nih.gov/search/asearch.asp. 

Office of Human Subjects Research, NIH — This office provides 
free computer-based training and certification on the use of human 
subjects in research. See http://ohsr.od.nih.gov. 

UCLA Online Training Portal — A source for online training for 
using human subjects in social and behavioral research. See http: 

/ / training.arc.ucla.edu. 

U.S. Department of Education — Information about protection of 
human subjects in research. See http://www.ed.gov/about/offices/list 
/ocfo/humansub.html. 
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STEP 5 

OBTAIN APPROPRIATE 
CONSENTS TO CONDUCT 
THE EVALUATION 



Step 5 involves meeting the requirements for obtain- 
ing consent as required by an IRB for research. The 
project director and the evaluator must obtain permission 
for subjects’ participation as well as informed (sometimes 
called “active”) consent and waivers of informed consent. 
They must also appropriately maintain anonymity and 
confidentiality for participants. 

OBTAINING PERMISSION FOR PARTICIPATION 

Federal regulations require that all participants in 
a research study consent to take part. They must be 
provided the opportunity to decide freely whether to 
participate — unless the research study uses only curricu- 
lar-based tests given in the course of teaching (e.g., math 
and reading tests). Moreover, if the student is a minor 
and the research is supported by the U.S. Department of 
Education, then the parents also must have the opportu- 
nity to allow or not allow the child’s participation. The 
two types of consent from students, parents and teachers 
are illustrated in Exhibit 1 1 . 



The terms active consent and passive consent are 
sometimes heard in discussing evaluations. The Federal 
Policy for the Protection of Human Subjects, or Common 
Rule provisions, use the term informed consent for active 
consent, and allow IRBs to waive informed consent under 
some conditions for minimal risk studies (34 CFR 97; 
also see USED/GPOS 2005b). In that case, the waiver 
can allow what is popularly referred to as passive consent. 
For frequently asked questions about this issue, see the 
NIH Web site http://grants.nih. gov/grants/policy/hs 
/ faqs_applicants.htm. 



EXHIBIT 11 

TYPES OF CONSENT THAT MUST BE 
OBTAINED FROM STUDY PARTICIPANTS 


TYPE OF 
CONSENT 


REOUIREMENTS 


PARTICIPANTS 


Waiver of 


Inform 


Teachers 


Informed 


participants 


Parents 


Consent 


by letter about 




the study and 


Students 18 




request that 
they return the 


or older 




accompanying 


Students 




form only 


younger than 




if they do 


18 (parental 




not wish to 


notification is 




participate. 


needed) 


Informed 


Participants 


Teachers 


Consent 


must give 






written consent 


Parents 




to participate 


Students 18 




in the study 


or older 

Students 
younger than 
18 (parental 
notification is 
needed) 



Obtaining informed consent, as distinguished from 
a waiver of informed consent, is preferred. A signed form 
or another written affirmation definitively establishes 
informed consent. A waiver of informed consent provides 
permission by default — that is, consent simply by not 
saying no. Informed consent from parents will be required 
by an IRB in most cases of school-based research involv- 
ing students. 

Project directors should be aware that informed-con- 
sent procedures have both budget and timeline implica- 
tions. Baseline data on human participants cannot be 
collected until after informed consent is obtained, which 
often can take six to eight weeks to acquire (Sherblom 
2004). The costs associated with acquiring informed 
consent can range from the cost of postage for mailing 
consent forms to parents to the cost of staff time to reach 
parents who require multiple, individual follow-up con- 
tacts before they will return the consent forms. 

While parents must consent to have their children 
participate in research, the students themselves are en- 
couraged to assent to participate. It is important to make 
clear to both parents and students that all participation is 
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voluntary and that no penalty can result from declining to 
participate in research. 

To decide whether or not to consent, participants and 
the parents of minor students must receive enough informa- 
tion about the evaluation to make an informed choice. The 
letter explaining the project can be sent on official school or 
district stationery and should include the elements outlined 
in Exhibit 12. Appendix C contains sample letters related to 
obtaining both informed consent and a waiver of informed 
consent. Letters of consent are also subject to IRB approval 
and must be included in the IRB application. 




MAINTAINING ANONYMITY 
AND CONFIDENTIALITY 

In addition to obtaining consent, both the school 
staff members and the evaluator must ensure that all 
participants are protected so their responses will not 
jeopardize them legally, emotionally or personally. 
Anonymity and confidentiality are two strategies for 
protecting the right of individuals to privacy and for 
easing any hesitation they may have about participating. 

Both confidentiality and anonymity assure partici- 
pants that any data they provide through surveys, assess- 
ment interviews or focus groups cannot be traced back 
to them. Confidentiality is the promise of the evaluators 
not to reveal any personal or identifying information, 



although this information is collected. Each subject is 
assigned a code number to protect his or her identity. 
Protection of confidentiality requires that these code 
numbers, or other indirect identifiers, not be used at any 
time to indicate personal or identifying information. 

In other words, the evaluators know the identity of the 
participants, but do not reveal it in their reporting. This 
strategy allows the evaluators to track the coded numbers 
(rather than individually named people) for attrition, 
participation and long-term outcomes. Although the 
evaluators can trace the coded number back to the 
participant, they follow protocols that maintain the 
person’s confidentiality (Posey, Davidson, and Korpi 
2003). 

With anonymity, however, names or code numbers 
are not used during the study so even the evaluators 
cannot identify a participant’s data. Anonymity is used 
to encourage participants to provide more honest and 
complete answers. The disadvantage of anonymity is that 
the evaluators cannot follow individuals over time to 
assess long-term outcomes or participant attrition. 

Character education evaluation protocols often 
involve the collection of information that participants 
consider sensitive (e.g., dishonest behavior, victimization, 
bigotry and problem behavior) . Even if the information is 
not sensitive, it is the responsibility of the project director 
and evaluator to ensure that data are never treated casu- 
ally. Procedures should be clearly articulated for keeping 
all evaluation data secure at all points in the collection, 
management, analysis, reporting and storage process. 
Procedures for secure storage or destruction and disposal 
of all data at the specified time after the end of the evalu- 
ation should be included in the IRB application. In some 
instances, the project director, the evaluator, or both may 
want to maintain and preserve data that have been col- 
lected and stored in a manner consistent with informed 
consent and IRB-approved methods so they can use it 
for further analysis or to inform future work on character 
education. Plans such as these should also be included in 
the IRB application. 



RESOURCES FOR ADDITIONAE INFORMATION 
ABOUT OBTAINING INFORMED CONSENT 
FROM STUDY PARTICIPANTS 



See Internet Resources on use and protection of human subjects at 
the end of Step 4 on page 29. 





STEP 6 

COLLECT AND MANAGE DATA 

Step 6 involves collecting and managing data, includ- 
ing (a) enlisting and maintaining the participation of 
support personnel, the intervention implementers, and 
control or comparison groups; (b) conducting pilot tests; 
and (c) creating and implementing a data management 
plan, which includes training the data collectors and 
monitoring data collection. 

ENLISTING AND MAINTAINING 
PARTICIPATION OF SUPPORT 
PERSONNEL, THE INTERVENTION 
IMPLEMENTERS, AND CONTROL OR 
COMPARISON GROUP STAFF MEMBERS 

The initial and ongoing commitment of district and 
school administrators is critical to the success of evaluat- 
ing any character education program. Schools are most 
likely to agree to participate and comply with evaluation 
design criteria if they (a) have a strong ongoing partner- 
ship with the evaluation team, (b) have confidence in the 
adequacy of the study to provide trustworthy answers 
to evaluation questions as well as in its feasibility, (c) 
believe that the study will lead to improvement in their 
school through implementation of the intervention, (d) 
believe that the intervention and its positive effects will 
be sustained beyond the research grant funding, and (e) 
hear from the project director and evaluator during the 
evaluation planning phase about efforts to minimize any 
disruptive impact of the study on the participating school 
(or schools). These considerations are especially important 
for control groups that may receive the intervention at a 
later date, should the study demonstrate effectiveness. 

Efficient data collection requires commitment from 
the intervention staff as well as school administrators 
and personnel. One way to obtain that commitment is 
to engage the school personnel and the implementers in 
helping to plan the logistics for data collection and man- 
agement. The school personnel — especially teachers — are 
likely to anticipate logistical problems that the project 
director, intervention staff members or the evaluator may 
not have realized. Having this information up front en- 
ables the evaluator to adjust the evaluation plan to avoid 
compromising the study. 

The project director is in a unique position to rein- 
force to school staff members the value of the evaluation. 
An important message to convey is that evaluation results 
can help school staff members improve their character 



education strategies, which may lead to further improv- 
ing student behaviors and academic performance. The 
evaluation is much more likely to succeed when schools, 
intervention staff members and the evaluation group have 
a sound relationship and a commitment to collecting data 
of high quality and usefulness. 

The project director and evaluator also should 
develop a strategy for maintaining the commitment of 
the control or comparison groups and for monitoring 
their activities so differences between the intervention and 
control conditions are documented and preserved over the 
course of the study. Adequate time and resources should 
be allocated to developing and maintaining a good work- 
ing relationship with the control or comparison group 
staff members. 

CONDUCTING A PILOT ROUND 
OF DATA COLLECTION 

A pilot round of data collection provides an opportu- 
nity to identify and correct any problems with the instru- 
ments or procedures before the evaluation begins. The 
pilot round helps the evaluation team to do the following: 

★ Estimate the amount of time required for inter- 
views, completing surveys and making observa- 
tions 

★ Determine whether participants can complete 
surveys without assistance from staff members 
or how much and what kind of assistance they 
will need 

★ Identify what data on school records are avail- 
able, complete and consistently maintained 

★ Determine whether instruments measure the 
same phenomenon and take account of likely 
differences that can be attributed to culture, 
development and reading levels 

★ Determine whether valid data can be obtained 
from instruments that have been translated into 
languages other than English 

Pilot rounds may vary; they may include some or 
all instruments and participants from only some or all 
groups. Sometimes piloting is not necessary. The evalu- 
ator, in consultation with the project director, should 
determine its necessity and how extensive it will be. 
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CREATING A DATA MANAGEMENT PLAN 

The evaluator should create a plan for monitoring 
data quality and the data collection process. If the data 
collectors have a plan for handling the data, they are 
better equipped to record and scan it for accuracy and 
completeness. 

The data management plan gives evaluators the 
information they need to access and understand the data 
easily. Often, evaluators use several types of data to assess 
a particular outcome. For example, a program description 
may state that one goal is to improve students’ prosocial 
behavior. Thus, before the intervention begins, evaluators 
might collect data on disciplinary referrals, might make 
in-class and out-of-class observations, and might conduct 
interviews with teachers, students and administrators, 
focusing on assessing prosocial behavior. These data may 
be collected at both the intervention and control or com- 
parison sites. The evaluators would immediately examine 
these different kinds of data to determine their usefulness 
for assessing prosocial behavior and would then decide 
which, or which combination, of them to use. 



RESOURCE FOR ADDITIONAL INFORMATION 
ABOUT COLLECTING AND MANAGING DATA 



U.S. Department of Education, Institute of Education Sciences 
(USED/IES). 2005. How to conduct rigorous evaluations of math and 
science partnerships (MSP) projects: A user-friendly guide for MSP proj- 
ect officials and evaluators. Prepared by the Coalition for Evidence- 
Based Policy, in partnership with the National Opinion Research 
Center of the University of Chicago. Washington, D.C.: USED/IES. 
See http://www.whatworkshelpdesk.ed.gov/sponsor.asp. 



TRAINING DATA COLLECTORS AND 
MONITORING THEIR WORK 

Before data collection begins, it is important to pro- 
vide formal training to everyone who will administer the 
data collection tools. The evaluator should prepare a data 
collection manual and go over the collection procedures 
in detail. The evaluator should hold a practice session 
during which the data collectors complete the instru- 
ments themselves and administer the instruments to one 
another. 

After data collection begins, the process should 
include frequent reviews of the data and meetings with 
the data collectors to ensure that they are following the 
procedures consistently and are progressing according to 
plan. The evaluator or evaluation team members should 
review completed instruments as they arrive to make sure 
each is correctly and fully answered. 




STEP 7 

ANALYZE AND INTERPRET DATA 

In Step 7, evaluators use processes that involve analy- 
ses and interpretation of results after the data have been 
collected, in addition to monitoring for common issues as 
each round of data is prepared for analysis. When analyz- 
ing data, the focus should be on intervention goals and 
evaluation questions. The evaluation should answer these 
basic questions: 

★ Did intervention participants demonstrate the 
desired levels or changes in knowledge, attitudes, 
beliefs, behaviors or some combination of these 
outcomes? 

★ Did the school demonstrate the desired level or 
change in its climate or culture (i.e., the school’s 
physical environment, safety, social atmosphere 
and lessening of discipline problems)? 

★ Were these observed levels and changes attribut- 
able to the character education intervention? 

★ How can the results and information gained 
from the intervention be used to guide practice? 

ANALYZING DATA ABOUT 
PROCESS OBJECTIVES 

The analysis plan outlines strategies for analyzing, 
summarizing and reporting data. The analysis plan can 
include a content analysis of narrative reports, particu- 
larly of interview data. The plan should state as precisely 
as possible how to code, summarize and report narrative 
data. If the design is quasi-experimental, then the analysis 
plan also should include dosage or intensity data — that is, 
how much of each intervention activity was done, how 
many people were involved, and how much of each activ- 
ity was administered to each participant for all outcome 
variables. In addition, analysis plans should include sum- 
maries of the number of times each participant engaged in 
each activity, the activity’s intensity (e.g., 15 minutes or 2 
hours), and the activity’s duration or frequency (e.g., one 
Saturday morning or twice a week for 16 weeks). Detailed 
plans lay out the specific data to be analyzed, thus ensur- 
ing that the evaluator analyzes the different kinds of data 
appropriately. 



ANALYZING DATA ABOUT 
OUTCOME OBJECTIVES 

Assuming that the evaluation design includes a 
control or comparison group, analyses will compare 
outcome objectives (results) from participants in the 
character education intervention with the same outcome 
objectives from those in the control or comparison group. 
Data analyses will assess the relationship of the interven- 
tion to the predicted effects as specified in the evaluation 
plan, and the evaluation plan should specify a general 
analytical approach. For example, if the evaluation team 
specified different outcomes for students, staff members 
and parents, then the analysis plan would specify separate 
procedures appropriate for assessing the data from 
each group. 

The evaluation design will also dictate appropriate 
methods for assessing the outcome data. For example, in 
quasi-experimental designs in which treatment and com- 
parison groups are selected using chosen criteria, analysis 
of intervention characteristics — such as training and 
dosage — is often appropriate and necessary. In contrast, in 
experimental designs, analysis of training and dosage is of- 
ten not appropriate. The evaluation plan also determines 
whether intermediate effects (e.g., the program affects 
school climate which then affects student outcomes) as 
well as final outcomes will be examined. 

MONITORING FOR ISSUES IN DATA ANALYSIS 

In general, the best way to prevent problems in the 
data is through careful planning during the proposal 
development phase and continuing teamwork throughout 
the project. Appendix D offers an Evaluation Checklist to 
assist the reader in that effort. However, even the best- 
laid plans can fall victim to unanticipated events that can 
affect the validity of a study. The evaluation design and 
team must be flexible if unavoidable changes in circum- 
stances arise and must carefully document the context of 
and reasons for these changes to ensure that findings can 
be interpreted appropriately. 

The project director should be aware of common 
issues that can negatively influence the soundness or 
validity of the study’s findings and, as mentioned in Step 
3, should work closely with the evaluator during the 
evaluation design process to specify procedures that will 
minimize the negative effect of these issues. The project 
director and evaluator should continue to work together 
during the data collection process to monitor procedures, 
and as each round of data is prepared for analysis, it 
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should be examined for evidence of each of the following 
common issues: 

Lack of intervention fidelity. The process evalua- 
tion should determine the fidelity of the intervention. In- 
tervention fidelity means that the program of intervention 
has been fully implemented as designed by the developers. 

Partial treatment and contaminated control and 
comparison groups. Partial treatment occurs when some 
groups engage in only part of the intervention because 
they drop out or are noncompliant. A similar problem can 
occur in control or comparison groups if for any reason 
they are exposed to or contaminated by any aspects of the 
intervention. The most valid way to address these issues 
is to use an intent-to-treat analysis. An intent-to-treat 
analysis requires that data from all participants who were 
randomly chosen or assigned to an intervention group 
be used when examining the effects of the intervention. 
Intent-to-treat analysis also requires that those data from 
participants who were assigned to control or comparison 
groups and who may have received some aspects of the 
intervention be analyzed along with the other data for 
those groups. Under these circumstances, such control 
or comparison groups are considered contaminated. The 
strength of intent-to-treat analysis is that it gives an- 
swers about whether the group that was targeted for the 
intervention, on average, benefited from it. These answers 
address policy-relevant questions with respect to the ben- 
efits, effectiveness and overall cost of an intervention. The 
problems of partial treatment and of control and compari- 
son group contamination are best solved by keeping in 
close contact with all groups and knowing what they are 
doing. 

Attrition. The loss of individuals (e.g., students, 
teachers, parents), classrooms or schools can threaten the 
evaluation design. Baseline data collected on participants 
or groups before they dropped out should be compared 
with the same data for those individuals and groups who 
remain in the study. Differences should be noted, and in 
the event the dropped individuals cannot be followed, 
the study’s results should be interpreted in light of the 
changed samples. 

Consent bias. In all studies, it is probable that a 
group of people will decline to participate and that some 
will not return the consent form at all. Those who do not 
participate may have different characteristics from the 
people who consent to take part. Consent forms should 
include a choice of declining and a request for minimal 
background information relevant to the study’s objectives. 
Differences between those who decline and those who 
participate should be noted. The best way to reduce this 



bias is to encourage everyone’s participation in the study 
and to conduct random assignment after consent has been 
obtained. 

Differential history of participants. Comparison 
groups should be chosen to match intervention groups as 
much as possible in terms of background characteristics. 
When participants can only be selected by nonrandom 
procedures (as in quasi-experimental design), the evalu- 
ator will need to use statistical techniques to account 
for the noncomparability between the intervention and 
comparison groups, but even in these cases, the evaluator 
cannot be sure that he or she has eliminated all effects of 
unknown factors on the outcomes. 

Design breakdown. The term design breakdown 
refers to the poor or incomplete execution of an evalua- 
tion plan. It includes problems such as the replacement 
of the original randomly chosen or selected schools or 
classrooms with different ones either at the time data 
collection begins or during the evaluation study; schools 
or classrooms that drop out of the study; failure to col- 
lect data in the time frame set by the evaluation plan; 
and failure to collect data appropriately (untrained data 
collectors, too little time to complete task, etc). To avoid 
design breakdown, the project director must work with 
the evaluator to ensure that the full evaluation plan is 
implemented as designed. 

Lack of measurement reliability. An unreliable 
measure is one that yields different responses depending 
on differences between interviewers or data collectors. Re- 
liable measures are stable; participants’ responses are not 
dependent on the interviewer or data collector. Results 
from reliable measures can be compared across differ- 
ent research studies. Lack of reliablity can be minimized 
by selecting measurement instruments with established 
reliability. If such measures do not exist, then the project 
director and evaluator may want to field-test instruments 
before using them in the actual evaluation study. 

Lack of measurement validity. A measure that 
lacks validity does not assess the outcome it is supposed to 
measure. A valid measure does assess what it is designed 
to measure, which allows for comparison of results across 
studies. Lack of validity can be minimized by using field- 
tested instruments that have demonstrated reliability and 
validity. In the early stages of designing the evaluation 
plan, it is important to select instruments that measure 
the kinds of outcomes the intervention is expected to 
produce. 

Response bias. The term response bias refers to the 
degree to which a self-report answer may not reflect reality 



because of the respondent’s misperception or deliberate 
deception. One type of response bias is social desirability, 
the tendency of individuals to give the answer that will 
provide the most favorable impression. A second type is 
instrument reactivity; that is, an effect that occurs when 
participants choose to respond differently than they 
normally would based on their perception of the intended 
goal of the instrument. Another form of response bias 
is item nonresponse, which occurs when the participant 
declines to answer certain questions. Finally, systematic 
bias occurs when treatment, control or comparison groups 
are more likely to answer certain kinds of questions than 
others. Evaluators can use multiple means to reduce the 
effects of response bias, including triangulation — the 
collection of data from three or more sources and the 
comparison of those who responded one way with those 
who responded differently to see whether they differ on 
demographic indices such as socioeconomic status, ethnic- 
ity and race, sex and age. Desirable response rates depend 
on the intervention and the participants, but generally, a 
70 percent or better response rate provides usable data. 

Contaminated or incorrect data ( values that are 
out of data range). Before the analysis stage begins, data 
should be checked to ensure that results will be as accurate 
as possible. For example, evaluators should thoroughly 
check the data for values that seem out of place (e.g., a 
child received services eight days in one week). 



RESOURCE FOR ADDITIONAL INFORMATION 
ABOUT ANALYZING AND INTERPRETING DATA 



U.S. Department of Education, Institute of Education Science 
(USED/IES). 2005. Reporting the results of your study: A user-friendly 
guide for evaluators of educational programs and practices. Prepared 
by the Coalition for Evidence-Based Policy, in partnership with the 
What Works Clearinghouse. Washington, D.C.: USED/IES. See 
http://www.whatworkshelpdesk.ed.gov/sponsor.asp. 



DISPLAYING RESULTS OF THE ANALYSES 

Although sophisticated statistical analyses are useful 
in evaluation, results are best displayed in clear, easy-to- 
understand charts and tables. Bar charts, pie charts, and 
simple tables often have the most effect on stakeholders, 
decision-makers, and even the scientific community. 
However, choosing the most appropriate vehicle through 
which to display results is key to expressing those results 
most effectively. Information can actually become more 
confusing if it is displayed in the wrong way. Appendix 
E provides guidance about the criteria to consider when 
choosing a particular type of display. 
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STEP 8 

COMMUNICATE 
EVALUATION RESULTS 



Developing and implementing an effective strategy 
for communicating evaluation results is extremely impor- 
tant. Simply increasing the quantity and the accessibility 
of information does not guarantee that stakeholders who 
are seeking knowledge will hnd it — or will find it useful. 
Accessing, absorbing and applying information requires 
a substantial investment of time, often in short supply 
among the project directors of character education inter- 
ventions. The evaluator should report the results to stake- 
holders and decision-makers in relevant and user-friendly 
terms (e.g., percentage of change or grade-level gain) so 
stakeholders can judge the educational significance. 

Moreover, to communicate results successfully, the 
project director and the evaluator must specifically relate 
information to all of the intervention’s various stakehold- 
ers. Possible avenues of communicating results include 
academic journals, newspapers, Web sites, formal reports, 
testimony to school boards and legislative bodies, and 
reports to parent-teacher organizations. With the excep- 
tion of publication in academic journals (and, sometimes, 
newspapers), the project director is usually in charge 
of dissemination. The director should consult with the 
evaluator, particularly to avoid overstating the evaluation 
findings. 

The content of each communication should be 
tailored to its audience because different aspects of the 
evaluation will interest some stakeholders more than 
others. What is communicated should depend on which 
information has the most meaning and value for a 
particular audience. Provide the most compelling 
information at the beginning of the presentation and state 
clearly any action that the specific audience should take 
based on the evaluation findings. 

Communication should include a coordinated set of 
media, interpersonal and community-based strategies to 
influence awareness, attitudes and knowledge about desir- 
able outcomes. The communication strategies and process 
can shape how the stakeholders use the evaluation results 
to make substantial and long-term decisions. The follow- 
ing guidelines may be helpful: 

★ Communication vehicles should be varied and 
include written information and electronic 
media that can be disseminated internally 
among the stakeholders and externally. 



★ Communications with stakeholders about results 
should build on a foundation of ongoing com- 
munications between the project director and 
stakeholders during earlier phases (i.e., before 
and during the intervention process). 

★ The project director and evaluator must ensure 
that the information communicated is accurate 
and meaningful. 



RESOURCES FOR COMMUNICATING 
EVALUATION FINDINGS 



Torres, R.T, H.S. Preskill, & M.E. Piontek. 2005. Evaluation strate- 
gies for communicating and reporting: Enhancing learning in organiza- 
tions. 2nd ed. Thousand Oaks, Calif.: Sage. 

Tufre, E.R. 1983. The visual display of quantitative information. 
Cheshire, Conn.; Graphics Press. 
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CONCLUSION 



Rigorous scientific evaluation of PCEP interventions 
is essential if character education is to secure a prominent 
and permanent place in our schools. Rigorous evaluation 
is the field’s best means to achieve the following: 

★ Acquire trustworthy information by which to 
continuously improve character education, thus 
advancing theories and knowledge of how pro- 
grams work and why they are effective 

★ Increase our understanding of how character 
education affects cognitive, emotional and social 
developmental processes of children and youths, 
thus enhancing theories of human development 

★ Increase our understanding of how to create 
effective collaborations among project directors 
and staff members, teachers and administrators, 
evaluators and community stakeholders, thus 
strengthening development, implementation and 
support for character education 

★ Demonstrate character education’s effective- 
ness to policymakers and decision-makers who 
can commit the necessary time and resources to 
the adoption and implementation of character 
education programs in K-12 schools 



Rigorously evaluating character education interven- 
tions is both possible and worthwhile. Success depends 
on careful planning, a strong stakeholder partnership, a 
collaborative team effort, and adequate resources. The 
U.S. Department of Education is pleased to offer this 
guide not only to the many current and future grantees 
funded under the Partnerships in Character Education 
Program but also to others who are embarking on the task 
of scientifically based evaluation of their character educa- 
tion projects. Rigorous evaluation will help to ensure that 
our young people and communities receive the benefit of 
interventions that have demonstrated effectiveness. 

As evaluation becomes a manageable task for collab- 
orative teams and leads to improved evaluation processes, 
character education programming and its outcomes will 
be enhanced. The vision is that effective character educa- 
tion programs will create healthier environments in our 
schools and communities — environments in which chil- 
dren can develop competencies, learn skills, and practice 
behaviors to become people of excellent character who are 
motivated to succeed personally, achieve academically and 
serve their communities. 
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APPENDIX A 



APPENDIX A: 
PERTINENT EEDERAL 
REGULATIONS 



This appendix outlines information on two federal 
regulations that are essential for project directors and 
evaluators: The Family Educational Rights and Privacy Act 
{FERPA) and the Protection of Pupil Rights Amendment 
{PPRA). 

The Family Educational Rights 
and Privacy Act (FERPA) 

Statute: 20 U.S.C. Section 1232g. Regulations: 34 
CFR Part 99. 

FERPA provides that an education agency or institu- 
tion, such as a local education agency (LEA), that receives 
U.S. Department of Education funds may not have a 
policy or practice of denying parents the right to do the 
following: 

★ Inspect and review their child’s education re- 
cords (34 CFR Section 99.10) 

★ Seek to amend their child’s education records 
(34 CFR Sections 99.20, 99.21 and 99.22) 

★ Consent to the disclosure of personally identifi- 
ahle information from their child’s education 
records except as specified by law (34 CFR 
Sections 99.30 and 99.31). The consent must 

(a) specify the records that may be disclosed, 

(b) state the purpose of the disclosure, and (c) 
identify the party or class of parties to whom the 
disclosure may be made. There are, however, 
certain specific exceptions to FERPA’s general 
consent rule, which will be discussed below. 

LEAs must annually notify parents and eligible 
students of their rights under FERPA (34 CFR Section 
99.7). These rights transfer to the student when he or 
she reaches the age of 1 8 years or attends a postsecondary 
educational institution at any age (“eligible student”). 

If the LEA or education institution under the LEA 
wishes to disclose “directory information” from educa- 
tion records, it is required by FERPA (34 CFR Section 
99.37) to notify parents and eligible students of the types 
of information it has designated as directory information 
and to provide an opportunity for the parent or eligible 
student to opt out of the disclosure of directory 
information. 



LEAs must also comply with FERPA’s redisclosure 
and recordation provisions, set forth in 34 CFR Sections 
99.32 and 99.33, except for disclosures that are specifi- 
cally exempted. 

As noted above, the general rule is that a parent or 
eligible student shall provide a signed and dated written 
consent before an LEA may disclose personally identifi- 
able information from education records; there are, how- 
ever, certain specific exceptions. FERPA permits LEAs 
to make disclosures, without consent, to the following or 
under the following conditions: 

★ School officials with a legitimate educational 
interest (as defined in annual notification) 

★ Other schools in which the student seeks or 
intends to enroll 

★ Federal, state and local educational authorities 
under certain conditions 

★ Organizations conducting studies on the school’s 
behalf, which the school has authorized, for 
certain purposes 

★ To comply with lawfully issued subpoenas or 
court orders 

★ Appropriate parties in connection with a health 
or safety emergency 

This list is a partial listing of the disclosures permit- 
ted under FERPA without consent. For guidance about 
specific circumstances involving the disclosure of person- 
ally identifiable information from students’ education 
records, school officials can contact the Family Policy 
Compliance Office (FPCO) by sending an e-mail to 
FERPA@ed.gov. FPCO’s Web site is http://www.ed.gov 
/ policy/ gen/ guid/fpco/ index.html. 

Protection of Pupil Rights Amendment (PPRA) 

Statute: 20 U.S.C. Section 1232h. Regulations: 34 
CFR Part 98. 

PPRA was amended by the No Child Lefi Behind 
Act of 2001 to give parents more rights with respect to 
the surveying of minor students and the collection of 
information from students for marketing purposes and for 
certain nonemergency medical examinations. Some of the 
requirements with respect to surveys are mentioned here. 
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In general, PPRA governs the administration to stu- 
dents of any “survey, analysis, or evaluation” that concerns 
one or more of the following eight protected areas, which 
covers “information concerning: 

1 . Political affiliations or beliefs of the student or 
the student’s parent 

2. Mental or psychological problems of the student 
or the student’s family 

3. Sex behavior or attitudes 

4. Illegal, anti-social, self-incriminating or demean- 
ing behavior 

5. Critical appraisals of other individuals with 
whom respondents have close family relationship 

6. Legally recognized privileged or analogous 
relationships such as those of lawyers, physicians 
and ministers 

7. Religious practices, affiliations or beliefs of the 
student or student’s parent 

8. Income (other than that required by law to de- 
termine eligibility for participation in a program 
or for receiving financial assistance under that 
program)” {PPRA, 20 U.S.C. Section 1232h). 

Local education agencies (LEAs) must provide 
parents and students effective notice of their rights under 
PPRA. 

Additionally, an LEA must “directly” notify, such as 
through the U.S. mail or e-mail, parents of students who 



are scheduled to participate in the administration of any 
survey containing one or more of the eight protected areas 
of information listed above, regardless of the funding of 
the survey. The notice must provide parents (a) with an 
opportunity to review the survey and (b) with an oppor- 
tunity to opt out of having their child participate in the 
survey. LEAs must obtain active consent and may not use 
a passive procedure (e.g., opting out by not responding) 
before a student is required to participate in such a survey 
that is funded in whole or in part with U.S. Department 
of Education funds. 

LEAs are also required to adopt policies — in con- 
sultation with parents — with respect to privacy issues, 
including the surveying of students, inspection of 
instructional material, and the administration of physical 
examinations or screenings. 

For further guidance about specific circumstances 
involving the administration of surveys or other require- 
ments in PPRA, school officials can contact FPCO by 
sending an e-mail to PPRA@ed.gov. Additional informa- 
tion is on the FPCO Web site: http://www.ed.gov/policy 
/ gen/ guid/ fpco/index.html. 

School officials can find a model of a notice and 
other helpful information related to PPRA and FERPA 
on the Web site of the Family Policy Compliance Office 
(FPCO) : http://www.ed.gov/ policy/ gen/guid/ fpco/ doc 
/pprasuper.doc. 
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APPENDIX B: 

OVERVIEW OE SCHOOL CLIMATE 
AND SCHOOL CULTURE 



School climate is a multidimensional idea encom- 
passing both objective characteristics of the school and 
perceptions of the school as a place to work and learn. 
Research on the influence of school climate on student 
performance and character has focused on various aspects 
and more often examined perceptions rather than objec- 
tive indicators. Because it is important for character edu- 
cation evaluation studies to consider school climate and 
the more specific idea, school culture, detailed definitions 
are offered here. School climate includes 

★ Physical, spatial and temporal characteristics 
related to building structure, size, location, and 
structure of space and time (e.g., schools within 
a school, classroom size and arrangements, and 
length of classes, etc.); 



School culture, another changeable aspect of school 
climate, which includes the values, traditions, norms, 
shared assumptions and orientations, and social expec- 
tations that express a school’s distinctive identity. Two 
particular aspects are 

★ Indicators of social systems, including student, 
teacher, staff and parent behavior within and 
among groups; school rules and policies; school 
safety; and relationships between the school and 
the community; and 

★ Perceptions of social expectations, including 
students’, teachers’, administrative staff’s and 
parents’ sense of trust and respect for one 
another; their sense of fairness of rules and 
policies and responsibility for upholding them; 
sense of school safety; sense of the school as 

a place of learning; expectations of student 
achievement; and feelings of school spirit 
or pride. 



★ Social characteristics related to a school’s profile, 
including percentage of students who receive free 
or reduced price meals; diversity of student body 
and staff; and teaching staff characteristics (e.g., 
male to female ratio, age profile, professional 
degrees and years of experience); 

★ Changeable characteristics related to a school’s 
profile, including school mission and goals; 
school leadership; performance indicators (e.g., 
grades and standardized test scores); safety (e.g., 
presence of security officers, police officers or 
both in or around school, and levels of violence 
and drug abuse); levels of prosocial behaviors; 
instructional materials and quality; and attrac- 
tiveness of halls and classrooms; and 

★ Changeable perceptions of students, teachers, 
staff and parents about the above three sets of 
characteristics. 
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APPENDIX C: 

SAMPLE LETTERS TO PARENTS (IN ENGLISH AND SPANISH) AND TO 
SCHOOL STALE MEMBERS AS WELL AS SAMPLE STUDENT ASSENT EORM 



The following letters are examples of informed consent letters that have been used in projects funded through the 
Partnerships in Character Education Program. Evaluators will need to customize these examples to fit their particular 
research design and the intervention context. Additionally, any consent form or letter concerning students must meet the 
requirements of PPRA, and school officials should be aware of these requirements. 
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SAMPLE LETTER TO PARENTS EOR WAIVER OE 
INEORMED CONSENT (PASSIVE CONSENT) 



[School Letterhead] 



[Date] 



Dear Parent or Guardian: 

[Number of schools] schools in [Name of school district] have been offered the opportunity to work 
with [University/Evaluator/Implementation Group Name] in implementing the [Name of project], designed 
to improve schools by providing a more caring environment for students. A partnership of school, home and 
community, [Name of project] emphasizes positive character traits by integrating them into everyday classroom 
activities. 

Our school is one of the schools selected to participate in this federally funded project. [Number of schools] 
of the schools are implementing the [Name of project] this year. The remaining schools will be implementing it in 
subsequent years. 

As part of the project, we need to collect information from your child. A voluntary survey will be 
administered in Feb. to all [Grade levels, for example, 4th, 8th, and 1 1th] graders at the grant schools. It will 
take only about 20 minutes to complete. The survey questions will focus on your child’s participation in school 
activities, his or her opinions about how students and teachers cooperate within the school, and his or her feelings 
toward school. 

In January, some parents may also receive a survey in the mail to complete. Should you receive one, it is 
important that you complete the survey and return it in the stamped envelope to the central location indicated, 
where it will be processed by an independent third party who will keep any identifying information confidential. 

The student and parent survey data will then be summarized along with staff information for use in program 
planning. All survey information will be compiled in statistical summary form only. No individual survey 
information will be used. 

A copy of the student survey is in the school office and available for you to examine. Should you prefer that 
your child not take the survey, simply contact the school. 



Sincerely yours. 



[Principal] 
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SPANISH VERSION OF SAMPLE LETTER TO PARENTS FOR 
WAIVER OF INFORMED CONSENT (PASSIVE CONSENT) 

MUESTRA DE LA CARTA A LOS PADRES DE FAMILIA 
PARA EL CONSENTIMIENTO PASIVO 



[Membrete de la Escuela] 



[Fecha] 



Estimados Padres o tutores: 

A [niimero de escuelas] escuelas en [nombre del distrito de la escuela] se le ha dado la oportunidad de 
participar con [Nombre del grupo de Universidad/Evaluador/implementacion] en la implementacion de [nombre 
del proyecto], que ha sido disenado para mejorar las escuelas que proporcionan un ambiente mas comprensivo a 
los estudiantes. [Nombre del proyecto] es una alianza entre la escuela, el hogar, y la comunidad que acentua los 
rasgos positivos del caracter, integrandolos en las actividades diarias del aula. 

Nuestra escuela es una de las escogidas para participar en este proyecto, lo cual es financiado por el gobierno 
federal. [Niimero de escuelas] de las escuelas aplicaran el [Nombre del proyecto] este ano. Las escuelas restantes lo 
haran en anos subsiguientes. 

Como parte del proyecto, necesitamos pedir informacion a su nino. Se llevara a cabo una encuesta 
voluntaria durante el mes de febrero para todos los estudiantes en los grados [por ejemplo, 4, 8, y 1 1] de las 
escuelas participantes. La encuesta tomara aproximadamente veinte minutos. Las preguntas de la encuesta se 
enfocaran en la participacion de su de nino en las actividades dentro de la escuela, sus opiniones sobre como los 
estudiantes y los maestros cooperan dentro de la escuela, y sus sentimientos hacia la escuela. 

Es posible que en enero algunos padres reciban tambien una encuesta por correo. Si la recibe, es importante 
que usted complete la encuesta y la devuelva en el sobre con franqueo pagado al lugar indicado, donde sera 
procesada por una entidad independiente que protegera sus datos personales. 

Se creara un resumen de los datos recibidos de los padres, los estudiantes y la informacion del personal escolar 
para asistir en la planificacion del programa. Toda informacion de la encuesta se proporcionara solamente en 
resumen estadi'stico. No se proporcionara ninguna informacion de encuesta individual. 

La escuela guarda una copia de la encuesta del estudiante que usted puede examinar. Si prefiere que su nino 
no tome la encuesta, simplemente comum'quese con la escuela. 



Atentamente, 



[Director] 
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SAMPLE LETTER TO PARENTS EOR INEORMED CONSENT 
(ACTIVE CONSENT) AND PARENTAL CONSENT EORM 



[School Letterhead] 



[Date] 



Dear Parent or Guardian: 

Your child has a wonderful opportunity to participate in an innovative program through the [School district 
name]. This year, your child’s school has chosen to be a part of the [Name of Project], [State purpose and activities 
of project.] 

The No Child Left Behind Act of 2001, which emphasizes “safe schools and strong character,” encourages just 
this type of educational program. President Bush has quoted Martin Luther King Jr., who said, “Intelligence plus 
character — that is the true goal of education.” This project is funded by the U.S. Department of Education under 
the Partnerships in Character Education Program. According to the Department, character education addresses 
themes such as caring, civic virtue and citizenship, justice and fairness, respect, responsibility, trustworthiness, and 
giving. 

We Need Your Help 

An integral part of the project is an evaluation of its effectiveness. [State project goals and research questions.] 
We need your permission for your child to participate in the evaluation research so we can measure the outcomes. 

Your child’s participation will involve the completion of a pretest survey at the beginning of the school year 
and a posttest survey at the end of the school year. Your child will complete this survey along with those in his or 
her entire class whose parents have given permission to participate. These surveys are available for you to read in 
the principal’s office. 

Each survey will take 30 to 45 minutes to complete. Your child’s teacher may also complete an observation of 
your child’s behavior at the beginning of the school year and again at the end of the school year. To study changes 
in student achievement and behavior, we will also be collecting student records, including grades, discipline 
records and standardized test scores. Finally, we will randomly select some students to participate in small 
discussion groups. All information collected in this study will remain confidential. 

Your child’s participation in this research study is completely voluntary. Your decision to allow your child to 
participate will not affect your child’s current or future relationship with his or her teacher, school or after-school 
program. You are free to withdraw your child from this study at any time. 

Questions You Might Have 

Why is this research being done? This study is being conducted to measure the effect of a classroom-based 
character education program on students and teachers. We will also evaluate the effectiveness of the interventions 
and, later, plan to expand the study to include your entire school and community. We want to ensure that we are 
providing an intervention that is beneficial to our students, so we are asking approximately [Number] students 
and their teachers to participate in this initial study during the [200X — 200Y] school year. 
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Sample Letter to Parents for Informed Consent (Active Consent) and Parental Consent Form (continued) 



What is the purpose of this research? We hope to determine that the [Name of project] will result in [Stated 
intended results such as increased student involvement in schools, increased awareness of character elements and 
themes, improved student behavior, and increased academic achievement] . If these results are achieved, then we 
will be able to share character education programs and resources used in the project with more schools in your 
district and throughout the country. 

What procedures are involved? There is no cost for your child to participate in this research. If you agree to 
your child’s participation, then he or she will complete two identical character education surveys. The first survey 
will be given in the fall, [Month, year] , and the second survey will take place in the spring, [Month, year] . In 
addition, your child may be asked to participate in a small group discussion. All information will be collected by 
[Name of project or office] staff members. 

Are there potential risks and discomforts? We do not anticipate any risks to your child as a result of participating 
in this study. Your child may feel slight discomfort responding to questions about citizenship, beliefs and practices 
in personal relationships, integrity, unlawful and antisocial behavior, honesty, ethical behavior, and respect for self 
and others. Students are not required to answer any questions that they do not wish to answer. 

What about privacy and confidentiality? Any and all information provided by your child will be kept 
confidential. All participants will be assigned an ID number for evaluation purposes. This number will not be the 
same as your child’s student ID number, and it will not be possible for anyone except the evaluator to identify 
your child’s name through use of this research ID number. The evaluator has promised not to reveal any personal 
or identifying information; thus, privacy and confidentiality of your child’s records will be preserved. 

What are the benefits of taking part in the research? The research collected for this study may improve the 
implementation of both the lessons and character education programs in your child’s classroom. This research 
will also inform and improve future implementation of schoolwide programs in your child’s school. Both you and 
your child can feel satisfaction in knowing that you are contributing to a study that will help us to develop better 
character education programs that will positively influence student behavior and academic achievement. 

Can I remove my child from the study? You can choose whether your child participates in this study. You may 
withdraw your child from this study at any time without consequences of any kind. 

Whom should I contact if I have questions? The. researcher conducting this study is [Evaluator’s name. 
Organization name] . If you should have any questions about this research study, you can contact [Evaluator’s 
name] by phone at [Phone number] or through e-mail at [E-mail address] . 

Remember; 

Your consent in this research is voluntary. You may choose to withdraw your child at any time. To allow your 
child’s participation, please sign the attached consent form and return it to your child’s teacher. 



Sincerely yours, 



[Principal] 
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Parental Consent Form 

I/We understand that staff members from the [Organization name] will 

conduct a research study in my/our child’s classroom. The purpose of the research study is to measure changes 
in students’ knowledge, attitudes and behaviors. As a part of this research study, my/our child may be asked 
questions about citizenship, beliefs and practices in personal relationships, integrity, unlawful and antisocial 
behavior, honesty, ethical behavior, and respect for self and others. 

1. I/We also understand that I/we have the right to inspect all survey instruments before they are administered 
to my/our child. Copies of the survey instrument and lesson samples may be reviewed in the principal’s of- 
fice. 

2. I/We hereby give permission for my/our child to participate in the [Name of 

Project] research study conducted by the [Name of school] . 



Date: 

Print Child’s Name 



Parent/Guardian Printed Name 



Parent/Guardian Signature 



Parent/Guardian Printed Name 



Parent/Guardian Signature 
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SPANISH VERSION OF THE SAMPLE LETTER TO PARENTS 
FOR INFORMED CONSENT (ACTIVE CONSENT) 

AND PARENTAL CONSENT FORM 

MUESTRA DE CARTA A LOS PADRES PARA EL CONSENTIMIENTO 
INFORMADO (CONSENTIMIENTO ACTIVO) Y PLANILLA 
DE CONSENTIMIENTO DE LOS PADRES 



[Membrete de la escuela] 



[Fecha] 

Estimados padres o tutores: 

Su nino tiene una gran oportunidad de participar en un nuevo e innovador programa a traves del [Nombre 
del distrito escolar] . Este ano, la escuela de su hijo participara en [Nombre del proyecto] . [Indique proposito y 
actividades del proyecto.] 

La Ley Que Ningun Nino se Quede Atrds del 2001, que enfatiza “la seguridad de las escuelas y el solido caracter 
de los estudiantes”, estimula este tipo de programa educativo. El Presidente Bush ha recordado una frase de 
Martin Luther King Jr., quien dijo, “La inteligencia mas el caracter — esa es la verdadera meta de la educacion”. 

Este proyecto esta financiado por el Departamento de Educacion de EE.UU. mediante el Programa Alianzas 
en la Ensenanza del Caracter. Segun el Departamento de Educacion, la ensenanza del caracter hace enfasis en 
temas tales como la solidaridad, virtud ci'vica y ciudadam'a, justicia e imparcialidad, respeto, responsabilidad y 
generosidad. 

Necesitamos su ayuda 

Una parte esencial del proyecto es una evaluacion de su eficacia. [Mencione las metas del proyecto y las 
preguntas de investigacion] . Necesitamos su permiso para que su hijo pueda participar en este estudio y para poder 
medir los resultados. 

La participacion de su nino incluira llenar una encuesta al principio del ano escolar y otra al terminar del 
ano. Su nino tomara esta encuesta junto con sus companeros de clase que ban recibido el permiso de sus padres 
para participar. Las encuestas estan disponibles en la oficina del director de la escuela de su hijo para que usted las 
pueda ver. 

Cada encuesta tomara de 30 a 45 minutos. El profesor de su nino tambien podn'a evaluar la conducta de 
su nino al principio y al fin del ano escolar. A fin de poder estudiar los cambios en los logros academicos y la 
conducta de los alumnos, tambien obtendremos los archivos de cada uno, incluidos las calificaciones, los archivos 
disciplinarios y los resultados en los examenes estandarizados. Finalmente, seleccionaremos al azar a algunos de 
los alumnos para que participen en pequenos grupos de discusion. Toda informacion colectada en este estudio 
permanecera confidencial. 

La participacion de su hijo en este estudio es completamente voluntaria. La decision que usted tome no 
afectara la relacion actual o futura de su nino con sus maestros, la escuela o con las actividades despues de las boras 
de clases. Usted puede separar a su hijo del estudio en cualquier momento. 

Preguntas que usted podria tener 

^Por que se estd haciendo este estudio? Este estudio se llevara a cabo con el proposito de medir en los estudiantes 
y los profesores el impacto del programa de ensenanza de caracter que se realiza en el salon de clase. Tambien 
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evaluaremos la efectividad de las intervenciones y mas tarde ampliaremos el estudio hack toda la escuela y 
la comunidad en general. En breve, queremos asegurar que estamos proveyendo un programa que beneficie 
a nuestros estudiantes. Para cumplir con este objetivo, estamos solicitando a aproximadamente [Numero] 
estudiantes y a sus profesores a participar en este estudio durante el ano escolar [200X-200Y] . 

^Cudl es el proposito de este estudio? Esperamos determinar si [Nombre del proyecto] resultara en [Mencione los 
resultados deseados tales como el incremento de la participacion de los estudiantes en la vida estudiantil, mayor 
conciencia sobre los elementos del caracter y un mejor desempeno academico]. Si se obtienen estos resultados, 
entonces podremos compartir los programas de la ensenanza del caracter y los recursos utilizados en el proyecto 
con mas escuelas en su distrito y en todo el pals. 

^Cudles son los procedimientos? No cuesta nada participar en el estudio. Si usted autoriza la participacion de 
su nino en este proyecto, el o ella llenara dos encuestas identicas sobre la ensenanza del caracter. La primera sera 
administrada en el otono [mes, ano] y la segunda en la primavera [mes, ano]. Adicionalmente, se le puede pedir a 
su hijo que participe en un pequeno grupo de discusion. Toda la informacion sera recolectada por el personal de 
[Nombre del proyecto] . 

^Existen posibles riesgos e incomodidades? No anticipamos ningun riesgo para su hijo como resultado de 
participar en el estudio. Su hijo puede sentir una ligera incomodidad al responder a preguntas sobre la ciudadam'a, 
las creencias y practicas en las relaciones personales, la integridad, la conducta antisocial e ilegal, la honestidad, la 
conducta etica y el respeto por si mismo y por otros. Los estudiantes no tienen que responder a ninguna pregunta 
a la cual no desean responder. 

^Que hay sobre la privacidad y confidencialidad? Toda. informacion provelda por su hijo permanecera 
confidencial. A todos los participantes se les asignara un numero de identificadon para los propositos de la 
evaluacion. Este numero sera distinto al numero de identificadon escolar de su hijo, y no sera posible que nadie, 
excepto el evaluador, identifique a su hijo a traves del codigo de investigacion. El evaluador ha prometido no 
divulgar ninguna informacion personal o que pueda identificar a su hijo. Esto garantizara la confidencialidad y 
privacidad de los documentos de su hijo. 

^Cudles son los beneficios de este estudio? Las investigaciones recopiladas en este estudio podrlan mejorar las 
lecciones en la clase y los programas sobre la ensenanza del caracter. Este estudio tambien informara y mejorara la 
futura implementacion de programas muchos mas amplios en la escuela de su hijo. Usted y su hijo pueden sentirse 
orgullosos al darse cuenta que estan contribuyendo a un estudio que nos ayudara a desarrollar mejores programas 
para fortalecer la conducta y el «ito academico. 

^Puedo retirar a mi hijo del estudio? Usted decide si su hijo participa o no en el estudio. Usted puede retirarlo 
del mismo sin ningun tipo de consecuencia. 

quien debo llamar si tengo alguna pregunta? El investigador que conducira el estudio es [Nombre del 
investigador y de la organizacion] . Si usted tiene alguna pregunta sobre el estudio puede comunicarse con 
[Nombre del evaluador] por telefono al [Numero de telefono] o puede enviar un mensaje a [direccion electronical. 

Recuerde: 

Recuerde que su participacion es voluntaria. Usted puede retirar a su hijo en cualquier momento. A fin 
de permitir la participacion de su nino, por favor firme el formulario de consentimiento adjunto y regreselo al 
profesor de su hijo. 



Atentamente, 



[Director] 
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Spanish Version of the Sample Letter to Parents for Informed Consent (Active Consent) and Parental Consent Form 
Muestra de carta a los padres para el consentimiento informado (consentimiento 
activo) y planilla de consentimiento de los padres (continued) 



Solicitud de consentimiento del padre 

Yo/Nosotros entendemos que miembros del personal de 

[Nombre de la Organizacion] conduciran un estudio en el salon de clase de mi/nuestro hijo. El proposito es 
medir cualquier cambio en el conocimiento, actitud y conducta de los alumnos. En este estudio se le podn'an 
bacer preguntas a mi/nuestro hijo sob re la responsabilidad como ciudadano, las creencias y practicas personales, la 
integridad, la conducta antisocial e ilegal, la honestidad, la etica, y el respeto por si mismo y por otros. 

1 . Yo/nosotros tambien entendemos que tenemos el derecbo de inspeccionar todos los materiales a utilizarse en 
la encuesta antes que scan administrados a mi/nuestro hijo. Una copia de la encuesta, as! como ejemplos de 
topicos a usarse, pueden ser revisadas en la oficina del Director de la escuela de mi/nuestro nino. 

2. Yo/Nosotros, por tanto, damos nuestro consentimiento para que nuestro hijo 

participe en el estudio de [Nombre del proyecto] conducido por [Nombre de la escuela] . 



Fecba 

Escriba el nombre del nino 



Escriba el nombre del padre o tutor 



Firma del padre o tutor 



Escriba el nombre del padre o tutor 



Firma del padre o tutor 
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SAMPLE LETTER REQUESTING CONSENT EROM SCHOOL 
STAEE MEMBERS EOR PARTICIPATION IN RESEARCH 



Note that this memo would be copied twice. The participants would sign the consent form in one copy and save the 
other copy for reference. The signed copy would be returned to the organization sponsoring the research. After receiving 
the signed copy from the participant, the researcher would then sign that copy. 



[Research organization letterhead] 



To: School Staff Member 
From; [Research organization] 

Re: Consent and permission for participation in research for [Name of project] 

You are being asked to participate in a research study to find out how to help students behave better and 
achieve more in school. This study is being conducted by [Researcher name] from the [Name of research 
organization]. You have been asked to participate because you are an employee of a school that is participating in the 
study. We ask that you read this form and ask any questions you may have before agreeing to be in the research 
study. 

Your participation in this research is voluntary. Your decision whether or not to participate will not affect 
your current or future relations with either your employer or the [Research organization] . If you decide to 
participate, you are free to withdraw at any time without affecting that relationship. 

Why is this research being done? This study is being done because we are interested in finding correlates 
or predictors of student character, social skills, behavior and academic achievement. To do this, we are asking 
students, parents, and school staff and administrators to answer [Number of surveys] surveys over the next 
[Number of years] years. During this time, we will be asking staff, students and parents from [Number of schools] 
[Geographic area name] elementary schools to complete the surveys. The staff survey should take about 30 
minutes to complete. 

What is the purpose of this research? If we are able to determine what affects student character, behavior 
and academic achievement, then we will be able to develop better programs that will help to decrease problem 
behaviors and increase academic achievement in our schools. 

What procedures are involved? If you agree to be in this research, we would ask you to fill out a total of 
[Number of surveys] surveys in [Number of years] years. The first survey is attached. The other surveys will be 
distributed in the same manner at the end of the next [Number of years minus 1]. You may complete this paper 
and pen version or a Web-based version of the staff surveys in your home or any other private location. It will take 
approximately 30 minutes to complete. 

What are the potential risks and discomforts? We do not anticipate any risks from participating in this survey. 
There is a possibility that you may feel some discomfort when answering the questions about substance use or 
violence. You do not have to answer any questions that you do not want to answer. 

What about privacy and confidentiality? The survey and your answers will be treated privately and 
confidentially, and the risk of breaking that confidentiality is minimal. All participants will be assigned an ID 
number for research purposes only. Any information that identifies individuals will not be released or published. 
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Sample Letter Requesting Consent From School Staff Members for Participation in Research ( continued) 



Are there benefits to taking part in the research? You will receive no direct benefits from your participation in 
this study. However, you may feel satisfied knowing that you are contributing to a study that will help us develop 
better programs for reducing school violence, decreasing substance use, and improving academic achievement. 

Will I he told about new information that may affect my decision to participate? During the course of the study, 
you will be informed of any significant new findings (either positive or negative) such as changes in the risks or 
benefits resulting from participation in the study or new alternatives to participation that might cause you to 
change your mind about continuing in the study. If new information is provided to you, then we will once again 
obtain your consent to continue participating in this study. 

What are the costs for participating in this research? There are no costs for your participation in this research. 

Can I withdraw or be removed from the study? You can choose whether or not to be in this study. If you 
volunteer to be in this study, you may withdraw at any time without consequences of any kind. You may also 
refuse to answer any questions you do not want to answer and still remain in the study. 

Whom should I contact if I have questions? The researcher conducting this study is [Evaluator’s name] . If you 
have question about this project, you may contact [Evaluator’s name] by phone [Phone number] or by e-mail 
[E-mail address]. If you have any questions about your rights as a research subject, you may call the local Office 
for Protection of Research Subjects [Phone number] . 

Remember; 

Your consent in this research is voluntary. You may choose to withdraw at any time. Your decision whether 
or not to participate will not affect your current or future relations with the university or your school. Whether 
or not you agree to participate, please sign one copy of the attached consent form and return it to your principal. 
Keep one complete copy (informational memo and consent form) for your records. 
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STAFF CONSENT AND PERMISSION FORM FOR PARTICIPATION IN RESEARCH 



[Name of Program] Research Project 

PLEASE RETURN THIS EORM TO YOUR PRINCIPAL. 

[Note that the second copy of this consent form would say PLEASE KEEP THIS FORM FOR YOUR RECORDS.] 



Signature of Subject 

I have read (or someone has read to me) the above information. I have been given an opportunity to ask 
questions, and my questions have been answered to my satisfaction. 

□ I AGREE to participate in this research. I have been given a copy of this form. 

D I DO NOT AGREE to participate in this research. I have been given a copy of this form. 



Signature of Subject 



Date 



Printed Name 



Signature of Research Staff Member 



Date 



Do NOT put this form with your survey. Return this form to your principal separately. 
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SAMPLE STUDENT ASSENT FORM 

(attached to research survey) 



[School Letterhead] 



PARTICIPANT ASSENT FORM 
[LEA or SEA Name] Character Education Study 



We are conducting a research study of students’ opinions about themselves, their school, and their 
community. This is a survey, not a test. There are no right or wrong answers. It is important that you answer each 
question honestly. The researchers from [Organization Name] are hoping to learn about students’ attitudes toward 
school and community involvement. The survey will be given in your classroom and will take about 1 5—20 
minutes to complete. 

You do not have to participate in the study, and you can stop participating at any time. You can skip a 
question if you do not want to answer it. If you decide not to participate, there will be no negative consequences. 
If you have any questions about the survey, please raise your hand, and the person giving the survey will help you. 
If you have any personal concerns about the survey, you can speak with a school counselor. 

Other than the researchers, no one — including students, teachers or your parents — will know your individual 
answers or be able to link your name with any of the research information. We will make every effort to keep your 
answers confidential. 



Name (please print) 



Signature 



Date 



Age 
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APPENDIX D 
CHECKLIST OE 
EVALUATION ACTIVITIES 



This checklist summarizes the steps to he taken as 
discussed in the Mobilizing for Evidence-Based Character 
Education guide. 

STEP 1: Partner with an evaluator and 
form an evaluation team. 

□ Find a skilled evaluator. 

□ If an outside evaluator is selected, then contract 
with that person or organization, following 
required policies and procedures for contracting. 

□ Assemble a collaborative advisory evaluation 
team that includes the program director, the 
evaluator and key stakeholders. 

□ Define roles and responsibilities for the proj- 
ect director and the evaluator (see Exhibit 1 , 
page 9). 

STEP 2; Develop a comprehensive program description. 

□ Develop the program description as part of the 
process to write the grant application proposal. 

□ Write a clear and comprehensive program 
description that is a collaborative effort between 
the project director and the evaluator. 

D Position the proposed program in relation to 
other character programs and relevant research 
in character education. 

D Determine the program goals for all involved 
stakeholders — students, teachers and the schools 
as well as administrators, parents and the 
community. 

□ Know the program requirements and features. 

□ Take into account school, district and commu- 
nity characteristics. 

□ Understand local, state and federal guidelines 
relevant to the intervention. 

□ Share the program description with key 
stakeholders. 



STEP 3; Prepare the evaluation plan. 

□ Collaborate in developing the evaluation plan 
and share with all stakeholders. 

□ Review character education program research, 
consider your own program goals and consult 
with stakeholders before writing evaluation 
questions. 

□ Understand both process and outcome evalua- 
tions, and decide what processes and outcomes 
will be evaluated. 

□ Write evaluation questions using the model 
worksheet (see Exhibit 2, page 16). 

□ For outcome evaluations, choose either an exper- 
imental or quasi-experimental research design. 

□ Decide sample size using a power analysis to aid 
in the decision. 

D Consider how to prevent or minimize threats to 
the validity of the evaluation research. 

□ Make a data collection plan that describes data 
sources, instruments and timelines (see Exhibits 
8 and 9, pages 24 and 25). 

STEP 4: Prepare and obtain Institutional 
Review Board approval. 

□ Understand the criteria used by an Institutional 
Review Board to determine whether an evalu- 
ation may be implemented (see Exhibit 10, 
page 28). 

□ Understand the requirements for conducting 
research with human participants. 

□ Submit the proposed evaluation research to an 
IRB for review and approval. 

□ Obtain a federalwide assurance (FWA) if the 
project will be engaged in nonexempt human 
subject research. 

□ Refer to EERPA and PPRA regulations to see 
whether they are applicable (see appendix A, 
page 43). 
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STEP 5: Obtain the appropriate consents 
to conduct the evaluation. 

□ Know the types of consent that must be ob- 
tained from study participants (see Exhibit 1 1 , 
page 31). 

□ Include all necessary content in letters requesting 
informed consent (see Exhibit 12 on page 32 
and appendix C on page 46). 

□ Maintain the anonymity, confidentiality or both 
of study participants. 

STEP 6: Collect and manage data. 

□ Enlist and maintain support and participation of 
personnel, implementers and evaluation research 
staff members. 

□ Conduct a pilot round of data collection. 

□ Create a data management plan. 

□ Train data collectors and monitor their work. 



STEP 7: Analyze and interpret data. 

□ Understand how to analyze data about process 
objectives. 

□ Understand how to analyze data about outcome 
objectives. 

n Continue to monitor for common problems as 
data are prepared for analysis. 

□ Display results in clear and easy-to-understand 
charts and tables (see appendix E, page 61). 

STEP 8: Communicate evaluation results. 

□ Communicate interim and final results to stake- 
holders. 

□ Tailor your message to the needs of each stake- 
holder group, but provide the context of the 
total study and results. 

□ Use a variety of communication strategies to en- 
sure that findings are presented clearly and that 
conclusions are solidly based on findings. 
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APPENDIX E 

EORMATS USED TO DISPLAY DATA RESULTS 



This appendix provides examples of formats frequently used to display data results from evaluating programs. Criteria 
to consider for using a particular format and key elements to include are accompanied by an example of that format. In 
addition to these examples, many other formats that clearly display results can also be used. 



COMPARISON BAR CHARTS 

Comparison bar charts visually highlight differences and similarities between groups at different points in time (see Ex- 
hibit E.l). Specific information about variables (such as groups and times) is shown along the horizontal axis of the graph, 
called the X axis. Groups and times would be defined in a legend, or small box, below the X axis. The vertical sideline, 
called the Y axis, indicates unit of measurement being used in the chart. The title of a chart should describe what it contains 
by using elements of the X and Y axes. Bars that are clustered together show a profile of several variables at one time. Be 
sure to clearly identify the unit of measurement and each variable shown. 



EXHIBIT E.1 

EXAMPLE OF A COMPARISON BAR CHART 



WHAT GRADE 7 STUDENTS THINK: 

RANKINGS OF THE IMPORTANCE OF CHARACTER TRAITS* 



5 

4.5 

4 

£ 3.5 

o 



S 2.5 
2 
1.5 
1 

Control PRE Control POST Experimental PRE Experimental POST 



■ Respect □ Integrity □ Civic-mindedness 



Source: Adapted by permission from Grove, 2004. 

Note: Control group (n = 312) comprised students from a middle school, surveyed on 
September 21 , 2004 (before, or PRE, intervention), and on May 15, 2005 (after, or POST, 
intervention). Experimental group (n = 485) comprised students from a middle school, surveyed 
on September 19, 2004 (before, or PRE, intervention), and on May 23, 2005 (after, or POST, 
intervention). 

* Each group was asked to rate on a scale of 1 (low) to 5 (high) the importance of the three 
character traits in the legend. 
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COMPARISON LINE GRAPHS 

Comparison line graphs can be used to highlight the changes in responses from different groups taken at different 
times (see Exhibit E.2). Comparison line graphs are used to show how the variable — in this case, the group response — 
changes from one time to another time. Each symbol (box, diamond, etc.) represents the score on the variable for one 
group at one time. A legend, or small box, below the chart defines groups represented by the symbols. The time span 
being shown is designated on the horizontal bottom line of the graph (X axis). The unit of measurement for the variable is 
defined along the vertical side line of the graph (Y axis). The lines link same symbols to show the change in the variable for 
each group from one time to another. Be sure to clearly define both axes and the symbols being used. 



EXHIBIT E.2 

EXAMPLE OF A COMPARISON LINE GRAPH 



HOW TEACHERS VIEW SCHOOL ENVIRONMENT BEFORE 
AND AFTER CHARACTER EDUCATION INTERVENTIONS 




Fall 2000 (PRE) Spring 2002 (POST) 



Experimental Control 



Source: Adapted by permission from Marshall, Caldwell and Owens, 2003 

Note: These results are for two matched schools. From the experimental school, 26 
teachers were surveyed before Intervention (PRE) and 12 teachers were surveyed after 
intervention (POST). From the control School, 20 teachers were surveyed PRE and 15 
teachers surveyed POST. Teachers were asked to rate on a scale of 1 to 5 the school 
environment on various items including support they received, collaboration for 
improvement and interpersonal relationships. Ratings were combined to form a factor 
with possible scores ranging from 0 (low) to 100 (high). 




^ 
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PIE CHARTS 

Pie charts are used to show proportions, either in terms of characteristics within group samples and populations or in 
terms of items or activities (see Exhibit E.3). The legend should identify what the full pie represents and what each wedge 
represents. Wedges should be easily distinguished from one another, even in a black and white printing design. Be sure to 
clearly label each pie wedge, including the specific proportion it represents. 



EXHIBIT E.3 

EXAMPLE OF A PIE CHART 



PERCENTAGE OF 57 SAMPLE SCHOOLS AT DIFFERENT LEVELS 
OF PARTICIPATION IN A NATIONAL OHARACTER EDUOATION PROGRAM: 2003 




Program 

Participation Levei 

□ No Activity 

□ Low Activity 

□ Medium Activity 

□ High Activity 



Source: Adapted by permission from Higgins-D’Alessandro et at, 2004. 

Note: The 57 schools that voluntarily participated in the study came from a total of 
200 schools invited to participate — 100 of which had been designated as medium-activity 
or high-activity schoois by the above character education program in 2000, and 1 00 of 
which were randomly chosen in 2001 from a pool of 800 schools that in previous years 
had adopted the same program. 
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RESULTS TABLES 

Results tables provide the variables that were measured, show specific results found, and indicate whether statistical 
significance was found for the results (see Exhibit E.4). 



EXHIBIT E.4 

EXAMPLE OF A RESULTS TABLE 



ANALYSIS OF DEGREE TO WHICH GRADE SCHOOL STUDENTS FELT A SENSE OF 
BELONGING IN RELATION TO DEGREE OF SUCCESS IN IMPLEMENTING A NATIONAL 
CHARACTER EDUCATION INTERVENTION IN THEIR SCHOOL 



BELONGING 

FACTOR 


LEVEL OF 
IMPLEMENTATION 


MEAN* 


STANDARD 

DEVIATION 


SIGNIFICANCE 

OF 

DIFFERENCES 
IN MEANS 


Students’ feelings 
of belonging 
(A = 468) 


High 


74.83 


11.21 


<.01 


Moderate 


72.74 


13.68 


Low 


67.62 


17.62 



Source: Adapted by permission from Marshall, Caldwell and Owens, 2004. 



Note: During spring 2003, an implementation survey was administered to certified staff in 22 
elementary schools participating in a national character education intervention. Schools were classified 
into three groups based on how well respondents rated the implementation: 7 were classified as high 
implementation; 1 1 as moderate; and 4 as low. Also in 2003, students in grades 3 and 4 of the same 
schools were surveyed on the sense of belonging they felt at school. For each school, a belonging factor 
based on survey responses was developed, and the mean was determined for each of the three groups 
of schools. 

* Means shown in italics are not significantly different {p a .05) from each other based on Tukeys 
honestly significant difference (HSD) Test. 

Exhibit Reads: These results indicate that there was no significant difference in feelings of belonging 
for students in high and moderate implementation schools; both groups of students were significantly 
more positive in their feelings of belonging than students in low implementation schools. 
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This glossary defines terms frequently used in evaluation. 



accountability: An obligation to accept responsibility 
and account for one’s actions. For education institutions, 
accountability means testing and evaluating to measure 
effectiveness in improving student achievement and in at- 
taining other educational purposes. 

affective: Relating to emotions, feelings or attitudes. 

analysis: Examination of a body of data and informa- 
tion using appropriate qualitative methods or statistical 
techniques to produce answers to evaluation and research 
questions. 

assent: The agreement by children younger than the age 
of 18 to be involved in a research study, requested after 
parental consent has been obtained. Children agree to 
participate by signing an assent form. 

assessment: Used as a synonym for evaluation. The term 
is sometimes restricted to approaches that consider or ex- 
amine a process or factor before an intervention is imple- 
mented, commonly referred to as a needs assessment. 

assurances: Signed forms that establish the obligation 
for an entity, such as a school district, to abide by 
federal regulations (e.g., for the protection of human 
participants). 

attrition: Loss of subjects from a study sample during the 
course of data collection; also called mortality. 

baseline: Data describing the condition or performance 
level of participants before intervention, treatment or 
implemented program. 

behavioral objectives: Measurable changes in behavior 
that an intervention is designed to achieve. 

benchmark: A point of reference or standard of behavior 
against which performance is compared. 

categorical variable: A variable whose values are simply 
categories and, therefore, cannot be quantified except 
by counting the number of cases in each category (e.g., 
counties or grade levels). 

character education: A learning process that enables 
students and adults in a school and community to under- 
stand, care about and act on core ethical values such as 
respect, justice, civic virtue and citizenship, and responsi- 
bility for self and others. 



coding: To translate a given set of data or items into de- 
scriptive or analytical categories for data labeling, sorting 
and retrieval. 

cognitive domain: The scope of knowledge as well as 
related skills and abilities that learners need to achieve 
various types of instructional objectives. 

cohort: A particular group in a study that has a statisti- 
cal factor such as age or membership in common. For 
example, the first cohort would be the first group to have 
participated in a training program. 

comparison group: In a quasi-experimental design, 
carefully chosen groups of participants who either do not 
receive the intervention or receive a different intervention 
from that offered to the primary intervention group. 

comparison group study: A quasi-experimental study 
that compares outcomes for intervention groups with 
outcomes for one or more comparison groups chosen 
through methods other than randomization. 

confidentiality: The protection of data and information 
from people other than those authorized to have access. 

conflict of interest: A situation in which the private 
interests of someone involved in the evaluation process 
(e.g., the interviewer, rater, scorer or evaluator) could or 
does have an effect, either positive or negative, on the 
quality of the evaluation activities, the accuracy of the 
data, or the results of the evaluation. 

consent bias: A skewing of the data and results that oc- 
curs when the requirement of explicit participant consent 
in an evaluation design results in the failure to capture 
the true characteristics of the target population in the 
sample under evaluation. 

contaminated data: Data that threaten the validity of 
an evaluation and can corrupt the outcomes through 
unintended influence (e.g., the control group adopts or 
receives the intervention being studied or another similar 
intervention) . 

control group: In an experimental design, a randomly 
selected group from the same population that does not 
receive the treatment or intervention that is the subject of 
the evaluation. 
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correlation: The degree of relationship between two 
variables, scores or assessments. Correlations, by them- 
selves, do not imply cause-and-effect linkages between 
two variables. 

criterion {sing.), criteria {pi.)-. A standard on which a 
judgment or decision can be based. In evaluation, out- 
comes are measured against this standard to determine 
whether success has been achieved on a variable. 

culturally sensitive relevance: The pertinence and 
soundness of evaluation methods, procedures or instru- 
ments when applied to particular cultures and population 
subgroups. 

data: Factual information that can be collected. Ex- 
amples of data include age, date of entry into a program 
intervention, reading level, and ratings or scores obtained 
from an instrument. Sources of data include case records, 
attendance records, referrals, assessment instruments and 
interviews. 

data-based decision-making: Using results from evalua- 
tion research as the basis for choosing an intervention. 

data collection instruments: Tools used to collect 
information for an evaluation, including surveys, tests, 
questionnaires, interview instruments, intake forms, case 
logs and attendance records. Instruments may be devel- 
oped for a specific evaluation or modified from existing 
instruments. 

data collection plan: A written document describing the 
specific procedures to be used to gather information or 
data. The plan describes who will collect the informa- 
tion, when and where it will be collected, and how it will 
be obtained. 

data display: A visual format for organizing information 
(e.g., graphs, charts, matrices or other designs). 

data reduction: A process of selecting, focusing, simpli- 
fying, abstracting and transforming data collected in the 
form of written field notes or transcriptions. 

data sources: The people, documents, products, activi- 
ties, events and records from which data are obtained. 

database: An accumulation of information, usually com- 
puterized, that is systematically organized for easy access 
and analysis. 

design: The process of creating procedures to follow in 
conducting an evaluation. 



design breakdown: A malfunctioning of the evaluation 
design, which threatens the validity of the evaluation and 
occurs as a result of an inadequately conceptualized or 
poorly executed evaluation design. 

desired outcomes: The results, defined in measurable 
terms, that an intervention, process, instructional unit or 
learning activity is designed to achieve. 

directory information: The type of information con- 
tained in a student’s education record, such as name, 
address, telephone listing, grade level, honors and awards 
and participation in officially recognized activities and 
sports that would not generally be considered harmful 
or an invasion of privacy if disclosed (34 CFR 99.3 and 
99.37; also see USED/GPOS 2005a). 

dissemination: The process of communicating informa- 
tion to specific audiences for the purpose of extending 
their knowledge, sometimes with the goal of modifying 
policies, practices or attitudes. 

dosage: How much of the intervention activity was done, 
how many people were involved and how much of each 
activity was administered to each participant, classroom 
or school over a specified length of time. 

effect size: Measurement of the strength of a relationship 
or the degree of change. 

effectiveness: The extent to which an intervention 
achieves its objectives. 

ethical evaluation: Evaluation that is designed and con- 
ducted in accordance with a moral code of conduct that 
respects and values the well-being of the implementer 
and the study’s participants, the good of the institution 
and its community, and the innate rights of individuals. 

evaluation: The process that provides accountability. A 
systematic method for collecting, analyzing and using 
information to identify effective and ineffective services, 
practices, and approaches. Generally speaking, evaluation 
is grouped in two broad categories — formative and sum- 
mative evaluation. 

evaluation plan: A written document that describes the 
overall approach or design that will guide the evaluation. 
The plan includes what evaluation will be done, how it 
will be done, who will do it, when it will be done, and 
the purpose of the evaluation. The plan is developed by 
the evaluator and project director after consultation with 
key stakeholders, and it serves as a guide for the 
evaluation team. 
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evaluator: An individual who is trained and experienced 
in designing and conducting evaluations and who uses 
tested and accepted research methodologies. 

evaluation team: A group of project staff members that 
includes, at minimum, the evaluator, the project director, 
and representatives of key stakeholders and that has the 
responsibility to oversee the evaluation process. 

evidence-based program: An intervention that has been 
evaluated scientifically and that has been found effective. 

experimental design: The random assignment of stu- 
dents, classrooms or schools to either the intervention 
group (or groups) or the control group (or groups) . Ran- 
domized experiments are the most efficient and reliable 
research method available for testing causal hypotheses 
and for making causal conclusions, that is, being able to 
say that the intervention caused the outcomes. 

experimental group: A group of individuals who receive 
the treatment or intervention that is being evaluated or 
studied. Experimental groups, also known as treatment 
or intervention groups, are usually compared to a control 
or comparison group. 

external evaluator: A person conducting an evaluation 
who is not employed by or closely affiliated with the 
organization conducting the intervention; also known as 
a third-party evaluator. 

fidelity: The extent to which an intervention or program 
is practiced and set forth as designed. It is one important 
focus of a process or formative evaluation. 

focus group: A group that is engaged by a trained 
facilitator in a series of discussions designed to elicit 
group members’ insights and observations on a topic of 
concern to the evaluation. The members of a focus group 
are selected because they share a common trait, interest, 
knowledge, attitude, or experience. 

formative evaluation: Sometimes known as process 
evaluation. See definition for process evaluation. 

goal: An ideal; a hypothesized, broadly stated outcome. 

A goal is reached by achieving a set of specific, measur- 
able objectives. 

immediate outcomes: Those changes in program partici- 
pants’ knowledge, attitudes or behaviors that occur dur- 
ing the course of an intervention. 

implementation fidelity: When evidence that is based 
on data shows that an intervention has been put into 
effect as intended. 



independent evaluator: An evaluator who is objective 
about the results of an intervention and who has no au- 
thority over program implementation or vested interests 
in the outcomes. 

informed consent: Permission to participate from 
parents representing minor children and agreement from 
other participants, which is provided through a signed 
form after those granting permission or agreement have 
received detailed information about the collection and 
use of evaluation data as well as the retention of or access 
to assessment data and information. 

Institutional Review Board (IRB): A committee or 
organization charged with reviewing and approving the 
use of human participants in research and evaluation 
projects. The IRB serves as a compliance committee and 
is responsible for reviewing reported instances of regula- 
tory noncompliance related to the use of human partici- 
pants in research. IRB approval is required for federally 
funded, nonexempt, human participants research. 

instrument: A device for collecting data — such as a 
survey, test or questionnaire — that can be used in process 
and outcome evaluations. (Also see definition of 
data-collection instruments in this glossary.) 

instrument reactivity: A reaction in which participants 
may modify their behavior based on their perception of 
the intended goal of the instrument, thus responding dif- 
ferently than they normally would. 

intent-to-treat analysis: A type of analysis that includes 
all randomized individuals in the conditions or groups to 
which they were originally assigned regardless of (a) the 
treatment they actually received, (b) their level of adher- 
ence, (c) their attrition, or (d) some combination of those 
factors. 

intermediate effects: Results of a program interven- 
tion or treatment that occur before the intended final 
outcomes. 

internal evaluator: A staff member or organizational 
unit who is conducting an evaluation and who is 
employed by or affiliated with the organization within 
which the project is housed. 

intervention: A program or innovation that is the subject 
of the evaluation. 
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logic model: A diagram showing the logic or rationale 
underlying a specific intervention. A logic model visually 
describes the link between (a) the intervention, require- 
ments and activities, and (b) the expected outcomes. It is 
developed in conjunction with the program theory. (Also 
see definition for program theory) 

longitudinal study: An investigation that follows a 
particular individual or group of individuals over a 
substantial period of time (three to five years is the norm 
today) to discover changes that may be attributable to the 
influences of the treatment or intervention. 

measurable terms: Describing project objectives in 
straightforward language that clearly states a specific area 
of knowledge, an attitude or a behavior that can 
be assessed. 

measure: {noun) An instrument or device designed 
to collect data that can be used to assess an outcome 
involving a change in quantity or quality of knowledge, 
skill level, attitude or behavior, such as student prosocial 
behavior, academic performance or community involve- 
ment. {verb) To determine or estimate the quality or 
quantity of change in knowledge, skill level, attitude or 
behavior identified as a desired outcome. 

methodology: The process, procedures and techniques 
used to collect and analyze data. 

norm-referenced: A scoring interpretation that defines a 
test score according to the performance of others on the 
same test. 

objective: A clearly identified, measurable outcome that 
leads to achieving a goal. The most straightforward 
method for stating objectives is by means of a specified 
percentage of increase or decrease in knowledge, skill, 
attitude or behavior that will occur over a given time 
period (e.g., by the end of the academic year, students 
will report demonstrating a 20 percent increase in caring 
behaviors toward their peers). 

observation protocols: The process through which 
trained individuals focus direct, systematic attention on 
key elements to gather information about the environ- 
ment or about behavior or demonstrations of knowledge, 
skills or attitudes. 

observer: A trained person who systematically collects 
evidence and makes notes about what is being observed 
in classrooms or other settings. The observer does not 
have to be an evaluator. 



outcome evaluation: An evaluation that assesses the 
extent to which an intervention affects (a) its participants 
(i.e., the degree to which changes occur in their knowl- 
edge, skills, attitudes or behaviors); (b) the environments 
of the school, community or both; or (c) both the partici- 
pants and environments as described in (a) and (b) . 

outcome objectives: The measurable changes in the par- 
ticipants’ knowledge, skills, attitudes, behaviors or in the 
school and community environment that are expected to 
occur as a result of implementing an intervention. 

outcomes: Measurable changes in (a) participants’ 
knowledge, skills, attitudes, and behaviors, or (b) in the 
schools and communities, that occur as a result of the 
delivered interventions. 

participants: Stakeholders who are engaged in project 
activities, including evaluation. 

percentile rank: A number indicating an individual’s 
performance score or attainment in relation to the distri- 
bution of scores of a representative group of individuals. 

A percentile rank of 95 means that the individual per- 
formed as well as or better than 95 percent of the group 
on which the percentile ranks are based. 

pilot test: {noun) A preliminary test or study of either a 
program intervention or an evaluation instrument, {verb) 
To conduct a preliminary study of an intervention or 
evaluation design to assess appropriateness of components 
or procedures and make any necessary adjustments. For 
example, an agency might pilot test new data-collection 
instruments developed for an evaluation. 

posttest: A test or measurement taken after a service or 
intervention has occurred. The results of a posttest are 
compared with the results of a pretest to seek evidence 
of the change in the participant’s knowledge, skills, at- 
titudes or behaviors or changes in schools or community 
environments that have resulted from the intervention. 

power analysis: A method used by the evaluation team 
to decide on the number of participants necessary to 
detect meaningful results. 

pre-post study: A study that involves administering the 
same measurement to study participants before and after 
the intervention to determine whether participants in 
an intervention change during the course of that 
intervention. 
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pretest: A test or measurement taken before a service or 
intervention begins. The results of a pretest are compared 
with the results of a posttest to assess change. A pretest 
can be used to obtain baseline data. 

process evaluation: A form of evaluation designed to 
determine whether the program is being or has been 
delivered as intended, sometimes referred to as formative 
evaluation. 

program evaluation: Research, using any of several 
methods, designed to test the influence or effectiveness of 
a program or intervention. 

program implementation activities: The intended steps 
identified in the plan for the intervention. 

program monitoring: The process of documenting the 
activities of program implementation. 

program theory of change: A statement of the as- 
sumptions about why the intervention should affect the 
intended outcomes. The theory includes hypothesized 
links between (a) the program requirements and activi- 
ties, and (b) the expected outcomes; it is depicted in the 
logic model (also defined in this glossary) . 

qualitative data: Nonnumeric data that can answer the 
how and why questions in an evaluation. These data are 
needed to triangulate (see definition in this glossary) 
results to obtain a complete picture of the effects of an 
intervention. 

qualitative evaluation: An evaluation approach that 
is primarily descriptive and interpretative. Qualitative 
methods are often used in process evaluation. 

quantitative data: Numerical information such as test 
scores and discipline records. 

quantitative evaluation: An evaluation approach that 
involves numerical measurement and data analysis based 
on statistical methods. 

quasi-experimental design: The nonrandom assignment 
of students, classrooms or schools to either the interven- 
tion group (or groups) or to the comparison group (or 
groups). Assignment may be based on matching or other 
selection criteria. Quasi-experiments cannot test causal 
hypotheses nor make causal conclusions. They identify 
correlations between the intervention and outcomes. 

random assignment: A procedure in which sample 
participants are assigned indiscriminately to experimen- 
tal or control groups, creating two statistically equivalent 
groups. 



random selection: A process by which participants are 
indiscriminately selected from a larger population, ensur- 
ing all subjects an equal chance of being chosen. 

random sampling: Selecting people or items from a 
larger population or group in a way that ensures every in- 
dividual or item has an equal probability of being chosen. 

randomization: Assignment of participants in the target 
population to intervention and control groups in a way 
that ensures every subject in the target population has 
the same probability to be selected for either group. 

randomized control trial: A study that indiscriminately 
assigns individuals or groups from the target population 
either to an intervention (experimental) group or to a 
control group to measure the effects of the intervention. 

recommendations: Suggestions that are derived from 
evidence-based findings and that propose specific actions. 

regression discontinuity: A quasi-experimental design in 
which participants are placed into treatment and control 
conditions based on a cutoff score on a quantitative as- 
signment variable such as a test score. 

reliability: The extent to which an instrument, test or 
procedure produces the same results on repeated trials. 

replicable: An attribute of assessment, observation 
system or evaluation, indicating that the process used 
to obtain the data and evidence is clearly stated and can 
be repeated. The term also refers to an intervention or a 
component of an intervention that can be repeated 
under conditions different from those of the original 
implementation. 

research-based: A descriptor indicating that an educa- 
tional intervention is grounded in research from psy- 
chology, education or other areas of scientific inquiry. 
Although the term was used previously to refer to an edu- 
cational intervention that had been scientifically evalu- 
ated and found to be effective, now, the terms evidence- 
based or science-based are preferred because these terms 
imply effectiveness rather than an academic inquiry. 

response bias: The degree to which a self-reported 
answer may not reflect reality because of the respondent’s 
misperception or deliberate deception. 

results: Relevant information gleaned from the informa- 
tion and data that have been collected and analyzed in 
an evaluation. 
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sample: A subset of a total population. A sample should 
be representative of the population because information 
gained from the sample is used to estimate and predict 
the population characteristics under study. 

school climate: Multidimensional aspects of a school 
encompassing both characteristics of the school and per- 
ceptions of the school as a place to work and learn. 

school culture: The values, traditions, norms, shared as- 
sumptions and orientations that give a school its distinc- 
tive identity. School culture includes the social systems 
and social expectations that affect all members. 

scientifically based research: Research that involves the 
application of rigorous, systematic and objective proce- 
dures to obtain reliable and valid knowledge relevant to 
education activities and programs. 

secondary data analysis: A follow-up analysis of data us- 
ing procedures to verify the accuracy of the results of the 
initial analysis or to answer questions different from the 
original questions. 

self-report measures: Instruments, usually surveys, 
through which individuals record their own recollections 
of behaviors, events, feelings, judgments and attitudes. 

single-subject study: A study that relies on the compari- 
son of treatment effects on a single participant or group 
of single participants. Findings based on this design 
are typically not considered to be generalizable to other 
members of the population. 

stakeholders: Individuals who have an interest in a 
project. Examples include students, teachers, the project’s 
source of funding, the sponsoring or host organization, 
internal project administrators, participants, parents, 
community members and other potential program users. 

standardized tests or instruments: Assessments, inven- 
tories, surveys or interviews that have been tested with a 
large number of individuals and have been designed to be 
administered to participants in a consistent manner. Test 
results of program participants on a particular standard- 
ized test can thus be compared to the test results of other 
populations on the same test. 

statistical significance: A general evaluation term refer- 
ring to the idea that a difference observed in a sample 
could not be attributed to chance. Statistical tests are 
performed to determine whether one group (i.e., the 
experimental group) is different from another group (i.e., 
the control or comparison group) on the measurable 
outcome variables used in a research study. 



student learning outcomes: Measures of student 
achievement in knowledge, skills, and other educational 
outcomes such as improved student attitudes and behav- 
iors. This term covers the acquisition, retention, applica- 
tion, transfer and adaptability of knowledge, attitudes 
and skills. 

summative evaluation: An evaluation conducted at the 
end of an intervention to determine whether an interven- 
tion achieved the intended outcomes. These evaluations 
can also be called outcome evaluations. 

transferability: The degree to which the knowledge and 
skills demonstrated in solving a problem related to a task 
can be used to solve other related problems and real- 
world activities. 

treatment group: Also called an experimental group, a 
treatment group is composed of a group of individuals 
receiving the intervention services, products or activities 
to be evaluated. 

triangulation: The multiple use of various sources of 
data, observers, methods and theories in investigations to 
verify an outcome finding. 

validation: The process of determining the validity of an 
instrument or evaluation study as defined below. 

validity: In terms of an instrument, the degree to which 
it measures what it is intended to measure, also described 
as the soundness of the instrument. In terms of an evalu- 
ation study, the degree to which it uses sound measures, 
analyzes data correctly and bases its inferences on the 
study’s findings. 

variable: An attribute of behavior, skill, quality or at- 
titude being studied or observed that is measurable. 

waiver of informed consent: Granting permission by de- 
fault (in other words, not refusing but also not providing 
explicit written consent) to participate in the collection, 
use, retention or access of data and information as part of 
a study or evaluation. 
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